Information processing method, device and computer readable storage medium

Abstract

The embodiment of the present application discloses an information processing method, device and computer-readable storage medium. The embodiment of the present application obtains a first file to be processed in a first preset format; parses the first file to be processed, and converts the first file to be processed into a second file to be processed in a second preset format; reads the number of feature dimensions and corresponding feature information of each training sample information in the second file to be processed, and stores each training sample information in a first preset memory device; reads each training sample information in the first preset memory device into a graphics processor for processing. In this way, by adding the number of feature dimensions to the training sample information, the corresponding feature information can be quickly determined through the feature dimension data, and only two reading operations are required to store each training sample information in the first preset memory device, and then transfer it to the graphics processor for processing, reducing information calculation and reading, and greatly improving the efficiency of information processing.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to an information processing method, device and computer-readable storage medium. Background Art

[0002] Current advertising and marketing systems generally use trained deep learning models to predict whether users have a need for a product, so that when the trained deep learning model predicts that a user has a need for a product, advertisements can be delivered to users who have a need for the product.

[0003] For example, the deep learning model for advertising marketing is trained using the HugeCTR framework, which is a GPU distributed training framework that mainly targets the recommendation click-through rate (CTR) scenario and supports distributed training and evaluation of large-scale sparse training data.

[0004] In the process of research and practice of the prior art, the inventors of the present application found that in the prior art, when faced with massive training data and a massive number of features, multiple input / output (I / O) operations are required, which will lead to a waste of computing resources and low efficiency of information processing. Summary of the invention

[0005] The embodiments of the present application provide an information processing method, device, and computer-readable storage medium, which can improve the efficiency of information processing.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] An information processing method, comprising:

[0008] Acquire a first file to be processed in a first preset format;

[0009] Parsing the first to-be-processed file, converting the first to-be-processed file into a second to-be-processed file in a second preset format, wherein the second to-be-processed file includes a plurality of training sample information, wherein the training sample information includes a number of feature dimensions generated based on feature dimension statistics and corresponding feature information;

[0010] Reading the number of feature dimensions and corresponding feature information of each training sample information in the second to-be-processed file, and storing each training sample information in a first preset memory device;

[0011] Each training sample information in the first preset memory device is read into the graphics processor for processing.

[0012] An information processing device, comprising:

[0013] An acquiring unit, configured to acquire a first file to be processed in a first preset format;

[0014] a parsing unit, configured to parse the first to-be-processed file and convert the first to-be-processed file into a second to-be-processed file in a second preset format, wherein the second to-be-processed file includes a plurality of training sample information, wherein the training sample information includes a number of feature dimensions generated based on feature dimension statistics and corresponding feature information;

[0015] A storage unit, used for reading the number of feature dimensions and corresponding feature information of each training sample information in the second to-be-processed file, and storing each training sample information in a first preset memory device;

[0016] The reading unit is used to read each training sample information in the first preset memory device into the graphics processor for processing.

[0017] In some embodiments, the storage unit is used to:

[0018] Read the number of feature dimensions of each training sample information in the second to-be-processed file;

[0019] Determining the amount of data to be read according to the number of characteristic dimensions;

[0020] The feature information corresponding to the number of feature dimensions is read based on the read data volume, and each training sample information is stored in a first preset memory device. In some embodiments, the reading unit includes:

[0021] A random subunit, used for performing random processing on each training sample information in the first preset memory device;

[0022] A storage subunit, used for sequentially selecting training sample information from the first preset memory device after randomization and storing it in a second preset memory device;

[0023] The reading subunit is used to read each training sample information in the second preset memory device into the graphics processor for processing.

[0024] In some embodiments, the second preset memory device includes a first queue and a second queue, and the storage subunit is used to:

[0025] Selecting a first identifier from a first queue in a second preset memory device;

[0026] Determine a first target memory block corresponding to the first identifier;

[0027] Sequentially selecting training sample information from the first preset memory device after randomization and storing it in the first target memory block;

[0028] When it is detected that there is no remaining storage space in the first target memory block, the first identifier is placed from the first queue to the second queue.

[0029] In some embodiments, the reading subunit is used to:

[0030] Selecting a second identifier from a second queue of the second preset memory device;

[0031] Determine a second target memory block corresponding to the second identifier;

[0032] Transferring the training sample information in the second target memory block to an image processor for processing;

[0033] When it is detected that no training sample information exists in the second target memory block, the second identifier is put from the second queue into the first queue.

[0034] In some embodiments, the acquisition unit is used to:

[0035] Scanning the storage device based on a preset storage path;

[0036] The scanned file is processed in a first preset format to obtain a first to-be-processed file in the first preset format.

[0037] In some embodiments, the acquisition unit is further used to:

[0038] The first file to be processed in the first preset format is downloaded through a distributed file system.

[0039] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for a processor to load to execute the steps in the above-mentioned information processing method.

[0040] A computer program product or a computer program, wherein the computer program product or the computer program comprises computer instructions, wherein the computer instructions are stored in a storage medium. A processor of a computer device reads the computer instructions from the storage medium, and the processor executes the computer instructions, so that the computer performs the steps in the above-mentioned information processing method.

[0041] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above-mentioned information processing method when executing the computer program.

[0042] The embodiment of the present application obtains a first file to be processed in a first preset format; parses the first file to be processed, and converts the first file to be processed into a second file to be processed in a second preset format; reads the number of feature dimensions and corresponding feature information of each training sample information in the second file to be processed, and stores each training sample information in a first preset memory device; reads each training sample information in the first preset memory device into a graphics processor for processing. In this way, by adding the number of feature dimensions to the training sample information, the corresponding feature information can be quickly determined through the feature dimension data, and only two reading operations are required to store each training sample information in the first preset memory device, and then transferred to the graphics processor for processing. Compared with the prior art that requires frequent information reading operations, the embodiment of the present application can reduce information calculation and reading, greatly improving the efficiency of information processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 is a schematic diagram of a scenario of an information processing system provided by an embodiment of the present application;

[0045] Figure 2a It is a flowchart of the information processing method provided in the embodiment of the present application;

[0046] Figure 2b The storage format of training sample information for related technologies;

[0047] Figure 2c The architecture diagram of the HugeCTR framework for related technologies;

[0048] Figure 3 is another flowchart of the information processing method provided by an embodiment of the present application;

[0049] Figure 4a A schematic diagram of the structure of an open source cluster computing framework provided in an embodiment of the present application;

[0050] Figure 4b A schematic diagram of the architecture of the information processing method provided in the embodiment of the present application;

[0051] Figure 4c A schematic diagram of a scenario of an information processing method provided in an embodiment of the present application;

[0052] Figure 4dA schematic diagram of another scenario of the information processing method provided in an embodiment of the present application;

[0053] Figure 4e A schematic diagram of another scenario of the information processing method provided in an embodiment of the present application;

[0054] Figure 4f A schematic diagram of another scenario of the information processing method provided in an embodiment of the present application;

[0055] Figure 5 is a schematic diagram of the structure of an information processing device provided in an embodiment of the present application;

[0056] Figure 6 It is a schematic diagram of the structure of the server provided in the embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0058] Embodiments of the present application provide an information processing method, device, and computer-readable storage medium.

[0059] See also Figure 1 , Figure 1 The scenario diagram of the information processing system provided for the embodiment of the present application includes: a terminal, and a server (the information processing system may also include other terminals in addition to the terminal, and the specific number of terminals is not limited here), the terminal and the server may be connected through a communication network, and the communication network may include a wireless network and a wired network, wherein the wireless network includes a wireless wide area network, a wireless local area network, a wireless metropolitan area network, and a combination of one or more of a wireless personal network. The network includes network entities such as routers and gateways, which are not shown in the figure. The terminal can exchange information with the server through the communication network. For example, when the terminal is running an application containing various types of push information, such as video, short video, microblog, and advertising applications, the terminal can collect the first to-be-processed file corresponding to the user characteristics and send it to the server.

[0060] The information processing system may include an information processing device, which may be integrated in a server. The server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Figure 1 In the method, the server is mainly used to obtain a first file to be processed in a first preset format; parse the first file to be processed, convert the first file to be processed into a second file to be processed in a second preset format, the second file to be processed contains multiple training sample information, the training sample information is composed of the number of feature dimensions and corresponding feature information generated based on feature dimension statistics; read the number of feature dimensions and corresponding feature information of each training sample information in the second file to be processed, and store each training sample information in a first preset memory device; read each training sample information in the first preset memory device into a graphics processor for processing. After deep learning, the server can push more accurate push information according to different users' terminals.

[0061] The information processing system may also include a terminal, which can install various applications required by users, such as video, short video, microblog, advertising and other applications. For example, the terminal can collect the first file to be processed corresponding to the user characteristics and send it to the server, and can also receive push information returned by the server.

[0062] It should be noted that Figure 1 The scenario diagram of the information processing system shown is merely an example. The information processing system and scenario described in the embodiments of the present application are intended to more clearly illustrate the technical solution of the embodiments of the present application, and do not constitute a limitation on the technical solution provided in the embodiments of the present application. A person of ordinary skill in the art will appreciate that with the evolution of information processing systems and the emergence of new business scenarios, the technical solution provided in the embodiments of the present application is equally applicable to similar technical problems.

[0063] The following are detailed descriptions of each.

[0064] In this embodiment, the description will be made from the perspective of an information processing device, which can be specifically integrated into a server having a storage unit and a microprocessor installed therein and having computing capabilities.

[0065] See also Figure 2a , Figure 2a : is a flow chart of an information processing method provided in an embodiment of the present application. The information processing method includes:

[0066] In step 101, a first to-be-processed file in a first preset format is obtained.

[0067] In order to better illustrate the embodiments of the present application, Figure 2b and Figure 2c For reference, Figure 2b The storage format of training sample information for related technologies, Figure 2c The following is an architecture diagram of the HugeCTR framework of the related technology. In the related technology, HugeCTR is a fast advertising coarse-ranking model training solution. Currently, it can only support the storage format of sparse matrices (Compressed Sparse Row, CSR), which can realize the distributed training and evaluation of large-scale sparse training data. In advertising training, data parsing and processing are important links in the entire training. The parsing and processing mainly converts data samples such as string type data (string), floating point data type (float) and 64-bit integer data type (int64) into hash indexes. The hash index is implemented based on the hash table, and only queries that match all columns are valid. For each row of data, the storage engine calculates a hash code for all index columns. The hash code is a small value, and the hash codes calculated for rows with different key values ​​are also different. The hash index stores all hash codes in the index and saves pointers to each data row.

[0068] It can be seen that parsing and processing are the upstream data producers of HugeCTR. The performance of this process will directly affect the entire training process. Figure 2c As shown, each training sample information 11 needs to be written into a specific command file, such as the file_list.txt file. The file path is obtained by reading the file_list.txt file through multi-threaded file reading 12, and multiple files to be processed are obtained. Then, the training sample information in each file is parsed through multi-threading 13. The threads may include 4, such as id_0 and id_1, and stored in batch (channel). When the number of training sample information accumulates to a batch maximum number, the batch maximum number may be 100, and the training sample information in the batch is placed in the memory block of the memory device, such as Figure 2cAs shown, there may be i_0 memory blocks and i_1 memory blocks, and so on. Each memory block is distributed in a corresponding queue, such as queues n_0 and n_1. The cumulative number of threads (i) and the number of thread IDs can be calculated to find the remainder for memory mapping 14, and the memory queue 15 into which each thread is placed is calculated, wherein each queue is bound to the corresponding thread, and each queue uses a lock to read and write and protect the data in the memory block. The memory block can be 32, and finally the graphics processing unit (GPU) 16 takes the data in each memory block from each memory queue 15 in sequence according to the ID sequence of the memory queue 15 and copies it. If a memory block in a memory queue 15 has not yet stored the training sample information, it will wait and block until the training sample information is filled, which will cause the information processing to be very inefficient. It should be noted that the training sample information stored in the memory block must be in the CSR storage format. The CSR storage format is as follows Figure 2b As shown, there is a total of one label and four feature dimensions (i.e., height, age, gender, and education level). When a thread reads training sample information, it can only read one label or one feature representation at a time. Therefore, the related technology needs to perform reading operations repeatedly, occupying input / output resources, and the information processing efficiency is very poor.

[0069] In order to solve the above problems, the embodiment of the present application can first directly obtain the first file to be processed in the first preset format, and the first preset format can be a tfrecords file format. The tfrecords file is stored in binary and is suitable for reading large quantities of data in a serial manner. For the first file to be processed, we can write a program to save the ordinary first file to be processed in the tfrecords data format. The storage type of the tfrecords data format can include data types such as string, float, and int64. The specific data can include labels, features, and plaintext data corresponding to the features to achieve subsequent large-scale processing, avoiding the need to write the first file to be processed into a specific command file each time, which is cumbersome and wastes user time.

[0070] In one implementation, obtaining the first file to be processed in the first preset format includes: downloading the first file to be processed in the first preset format through a distributed file system.

[0071] Among them, the first file to be processed in a first preset format (i.e., tfrecords data format) can be downloaded through a distributed file system (Hadoop Distributed File System, HDFS), so there is no need to store the first file to be processed on a disk, and there is no need to start a separate thread to scan and read the file from the disk, thereby reducing the use of the central processing unit.

[0072] In step 102, a first file to be processed is parsed and converted into a second file to be processed in a second preset format.

[0073] The second preset format may be a CSR storage format, that is, the feature name may be parsed from the first to-be-processed file in the first preset format (such as the tfrecords file format), and the corresponding feature name, such as the plain text data corresponding to the feature name of height, may be hashed and converted into a corresponding hash key value (hash key), such as converting a character string of "male" for gender into a hash key value of 1. Similarly, the first to-be-processed file may be converted into a second to-be-processed file in the CSR storage format, and the second to-be-processed file may directly parse the hash key value and store it in a memory device, which may be a storage unit of a preset size opened in the storage space, such as a storage unit of 1024MB (megabytes).

[0074] It should be particularly noted that, unlike the form of one label and four feature dimensions in the CSR storage format of the related technology, the training sample information stored in the to-be-processed file in the embodiment of the present application is composed of the number of feature dimensions and the corresponding feature information generated by feature dimension statistics. The number of feature dimensions can be the sum of the number of labels, feature values ​​and feature names, for example 23. The number of feature dimensions can represent the subsequent entire feature information, that is, the label and subsequent feature representation can be directly obtained without repeated reading.

[0075] In some implementations, parsing the first file to be processed and converting the first file to be processed into a second file to be processed in a second preset format includes:

[0076] (1) parsing the first file to be processed to obtain corresponding feature names;

[0077] (2) when it is detected that the feature name matches a preset feature name stored in the preset feature table, obtaining the plain text information corresponding to the feature name;

[0078] (3) Hash the plaintext information to obtain the corresponding hash key value;

[0079] (4) Generate a second to-be-processed file in a second preset format according to the training sample information containing the hash key value.

[0080] Among them, since different deep learning models require different training sample information, a preset feature table can be pre-configured. The preset feature table (table) can store multiple feature names of the training sample information needed for the current deep learning. In this way, the first file to be processed can be parsed to obtain the corresponding feature name, and detect whether the feature name matches the preset feature name stored in the preset feature table, that is, the detected feature name is pre-stored in the preset feature table. When it is detected that the feature name matches the preset feature name stored in the preset feature table, it means that the feature name is the training sample information that needs to be trained by the current deep learning model. The plaintext information corresponding to the feature name can be directly obtained, and hash conversion can be performed to obtain the corresponding hash key value, for example, the feature name "undergraduate degree" is converted into a hash key value "1".

[0081] Furthermore, the training sample information containing the hash key value, that is, the training sample information that the current deep learning model needs to train, is generated into a second file to be processed in a second preset format (that is, CSR data format).

[0082] In some implementations, generating a second to-be-processed file in a second preset format according to the training sample information including the hash key value includes:

[0083] (1.1) Count the number of feature dimensions of each training sample information containing a hash key value;

[0084] (1.2) combining the feature information of each training sample information including the hash key value and the corresponding number of feature dimensions to obtain combined training sample information;

[0085] (1.3) Generate a second to-be-processed file in a second preset format based on the combined training sample information.

[0086] Among them, the number of feature dimensions containing each hash key value, that is, the number of training sample information required by the current deep learning model, can be counted. The number of feature dimensions includes feature identifiers and feature value numbers. The feature identifier is the number of feature names containing representations, and the feature value number is the total number of feature values. For example, Figure 2b As shown, the feature identifier can be 4, which is the number of labels + 5 age identifiers + 4 gender identifiers + 6 educational qualifications. The number of feature values ​​is 4 values ​​for height, 5 values ​​for age, 4 values ​​for gender, and 6 values ​​for educational qualifications. It can be concluded that the number of feature dimensions is 23.

[0087] In order to distinguish related technologies, the feature information of each training sample containing the hash key value can be Figure 2bThe feature information of the CSR of the related technology shown and the corresponding feature dimension number 23 are combined to obtain the combined training sample information. Since each combined training sample information contains the feature dimension number, the feature dimension number and the feature information represented by the feature dimension number can be directly read to realize the input / output operation of the training sample information, which greatly improves the efficiency of information processing.

[0088] In step 103, the number of feature dimensions and corresponding feature information of each training sample information in the second to-be-processed file are read, and each training sample information is stored in the first preset memory device.

[0089] Among them, the first preset memory device can be a storage unit of a preset size pre-opened in the storage space, such as a storage unit of 512MB (megabytes). Since each training sample information in the second to-be-processed file contains the number of feature dimensions that characterize the feature information, the number of feature dimensions of each training sample information in the second to-be-processed file can be read first, and the feature information corresponding to each to-be-trained information can be determined according to the number of feature dimensions for a second reading, so that each training sample information can be stored in the first preset memory device. Please refer to Figure 2b As shown, unlike the related art which needs to read 5 times to realize reading the training samples into the memory, the embodiment of the present application only needs to read twice to store each training sample information into the first preset memory device. When the amount of training sample information is huge, it can save orders of magnitude of data input / output operations, and the larger the feature dimension of the training sample information, the more obvious the effect, which greatly improves the efficiency of information processing.

[0090] In one embodiment, reading the number of feature dimensions and corresponding feature information of each training sample information in the second to-be-processed file, and storing each training sample information in the first preset memory device, may include:

[0091] (1) reading the number of feature dimensions of each training sample information in the second to-be-processed file;

[0092] (2) determining the amount of data to be read according to the number of feature dimensions;

[0093] (3) Reading feature information corresponding to the number of feature dimensions based on the read data volume, and storing each training sample information in a first preset memory device.

[0094] Among them, the number of feature dimensions of each sample information to be trained in the second file to be processed can be read. For example, the number of feature dimensions can be 23. Since the number of feature dimensions can represent the subsequent entire feature information, the number of feature dimensions can be used to determine that the amount of data to be read subsequently is 23 data. Then, based on the read data amount, the feature information corresponding to the number of feature dimensions can be directly read, so that the entire training sample information can be stored in the first preset memory device with only two reads.

[0095] In step 104, each training sample information in the first preset memory device is read into the graphics processor for processing.

[0096] Among them, the first preset memory device can include multiple memory blocks, for example, 50 memory blocks, each memory block can store 50 batches of samples, and the size of the batch can be set arbitrarily, for example, 100, that is, one batch can store 100 training sample information. In this way, each memory block in the first preset memory device can be traversed in sequence according to the preset order, and the training sample information in the memory block can be read into the GPU for processing, so as to realize distributed training and evaluation of training data. Since the number of input and output readings of training sample information is reduced, the utilization rate of GPU and CPU (central processing unit) can be saved, which greatly improves the efficiency of information processing.

[0097] In one embodiment, the step of reading each training sample information in the first preset memory device into the graphics processor for processing may include:

[0098] (1) performing random processing on each training sample information in the first preset memory device;

[0099] (2) selecting training sample information from the first preset memory device after randomization and storing it in the second preset memory device;

[0100] (3) Reading each training sample information in the second preset memory device into the graphics processor for processing.

[0101] Among them, the first preset memory device can be set as a shuffle memory, and the shuffle memory can implement the shuffle method, that is, each memory block in the shuffle memory and the training sample information in the memory block are randomly shuffled. In the related art, positive samples and negative samples are continuous, which will cause the trained deep learning model to be not robust and the recognition accuracy will be relatively low. In this way, the embodiment of the present application shuffles the distribution of positive samples and negative samples in the first preset memory device through the shuffle method, and sets a second preset memory device as an intermediate cache state. The second preset memory device can be a storage unit of a preset size pre-opened in the storage space, such as a 256MB (megabyte) storage unit.

[0102] Since the processing speed of the CPU and the processing speed of the GPU are inconsistent, for example, the CPU processes too fast, resulting in data squeezing, the GPU needs to wait for the storage of a certain memory block to be completed before traversing the next memory block, causing congestion. Therefore, the embodiment of the present application can set a cached second memory device to store the training sample information cached by the CPU. Correspondingly, the GPU can directly read the processed first training sample information from the cached second memory device for processing, avoiding the problem of congestion and further improving the efficiency of information processing.

[0103] In this way, by supporting the TFRecord format, feature engineering can be adapted (seamlessly adaptable), and the files in the TFRecord format can be improved to files in the CSR format, reducing the frequent reading times of IO data, thereby improving the utilization of the CPU and GPU. Secondly, the design of the memory block is improved to support more threads to parse data concurrently, occupying less memory, and at the same time preventing the problem of a single thread parsing being stuck and causing the entire process to be stuck. From the above, it can be seen that the embodiment of the present application obtains a first file to be processed in a first preset format; parses the first file to be processed, and converts the first file to be processed into a second file to be processed in a second preset format; reads the number of feature dimensions and corresponding feature information of each training sample information in the second file to be processed, and stores each training sample information in a first preset memory device; reads each training sample information in the first preset memory device into a graphics processor for processing. In this way, by adding the number of feature dimensions to the training sample information, the corresponding feature information can be quickly determined through the feature dimension data, and only two reading operations are required to store each training sample information in the first preset memory device, and then transferred to the graphics processor for processing. Compared with the prior art that requires frequent information reading operations, the embodiments of the present application can reduce information calculation and reading, greatly improving the efficiency of information processing.

[0104] The following is a further detailed description with examples.

[0105] In this embodiment, the information processing device is specifically integrated into a server as an example for description.

[0106] See also Figure 3 , Figure 3 Another flowchart of the information processing method provided in the embodiment of the present application. The method flow may include:

[0107] In step 201, the server scans the storage device based on a preset storage path, processes the scanned file in a first preset format, and obtains a first to-be-processed file in the first preset format.

[0108] The server in the embodiment of the present application may be a cloud server using cloud technology, and may integrate an open source cluster computing framework, such as the Spark engine or the Spark engine, which is a fast general computing engine designed for big data processing. Figure 4a As shown, Figure 4a A schematic diagram of the structure of an open source cluster computing framework provided in an embodiment of the present application.

[0109] The application layer A may include a package for structured data (Spark SQL), a component for stream computing (Spark Streaming), a library for machine learning (MLlib (machine learning)), and a tool set for graph operations and calculations (Graph X). The package for structured data is a package used by Spark to operate structured data. Through Spark SQL, SQL dialects can be used to query data. Spark SQL supports multiple data sources, such as data warehouse tools (Hive) tables, etc. The component for stream computing is a component provided by Spark for streaming real-time data, and an application programming interface (Application Programming Interface, API) for operating data streams is provided. The library for machine learning provides a library of common machine learning functions, including classification, regression, clustering, collaborative filtering, etc., and also provides additional support functions such as model evaluation and data import. The tool set for graph operations and calculations is a set of algorithms and tools for controlling graphs, parallel graph operations and calculations.

[0110] The core data computing layer B may include the code function layer (Spark Core) of the open source cluster computing framework, which implements the basic functions of Spark, including modules such as task scheduling, memory management, error recovery and storage system interaction. The Spark Core also includes the API definition of Resilient Distributed Datasets (RDD).

[0111] The resource scheduling layer C may include a local operation mode, an open source general resource management system (YARN), an open source distributed resource management framework (Mesos), etc., for resource management.

[0112] The data resource layer D may include a distributed file system (Hadoop Distributed File System, HDFS) or a distributed, column-oriented open source database (HBase), etc.

[0113] The Spark engine can be used to implement distributed data acquisition or training processing, providing efficient computing speed for processing data streams. Spark supports APIs in multiple development languages, allowing for rapid construction of different applications.

[0114] Among them, the first format is the tfrecords data format, the storage device can be a disk, and the preset storage path can be a disk path specifically used to store training sample information, such as D:\XL. The server can scan the storage device based on the preset storage path, parse the scanned file in the tfrecords data format, and obtain the first file to be processed in the tfrecords data format.

[0115] Please also read Figure 4b As shown, Figure 4b This is a schematic diagram of the architecture of the information processing method provided in an embodiment of the present application. The server scans the storage device based on a preset storage path through a single-threaded file name scan 11 to obtain a first file to be processed in a plurality of tfrecords data formats.

[0116] In step 202, the server parses the first file to be processed and obtains the corresponding feature name. When it is detected that the feature name matches the preset feature name stored in the preset feature table, the server obtains the plaintext information corresponding to the feature name, and performs hash conversion on the plaintext information to obtain the corresponding hash key value.

[0117] The feature names can be parsed from the first to-be-processed file in the first preset format (i.e., tfrecords file format), and the corresponding feature names, for example, can be referred to in conjunction with Figure 4c As shown, Figure 4c The storage format of the training sample information provided in the embodiment of the present application, the feature name may include height, age, gender and education level.

[0118] Since different deep learning models require different training sample information, a preset feature table can be pre-configured. The preset feature table (table) can store multiple feature names of the training sample information required for the current deep learning. For example, please refer to Figure 4d As shown, the preset feature table 31 may include features such as Age, height, Sexual, and educational background.

[0119] In this way, the first file to be processed can be parsed to obtain corresponding feature names, for example, the feature names are height, age, gender and education level, and it is detected whether the feature names match the preset feature names stored in the preset feature table (Table 1). When it is detected that the feature name has been stored in the preset feature table in advance, that is, when it is detected that the feature name matches the preset feature name stored in the preset feature table, it means that the feature name is the training sample information that needs to be trained by the current deep learning model. The plaintext information corresponding to the feature name can be directly obtained and hashed to obtain the corresponding hash key value, for example, the feature name education level "bachelor's degree" is converted into a hash key value "1".

[0120] In step 203, the number of feature dimensions of each training sample information containing a hash key value is counted, the feature information of each training sample information containing a hash key value and the corresponding number of feature dimensions are combined to obtain the combined training sample information, and a second to-be-processed file in a second preset format is generated based on the combined training sample information.

[0121] Among them, the number of feature dimensions containing each hash key value, that is, the number of training sample information required by the current deep learning model, can be counted. The number of feature dimensions includes feature identifiers and feature value numbers. The feature identifier is the number of feature names containing representations, and the feature value number is the total number of feature values. For example, Figure 4c As shown, the feature identifier can be 4, which is the number of labels + 5 age identifiers + 4 gender identifiers + 6 educational qualifications. The number of feature values ​​is 4 values ​​for height, 5 values ​​for age, 4 values ​​for gender, and 6 values ​​for educational qualifications. It can be concluded that the number of feature dimensions E is 23.

[0122] In order to distinguish related technologies, the feature information of each training sample containing the hash key value can be Figure 2b The feature information of the CSR of the related technology shown and the corresponding feature dimension number E23 are combined to obtain the combined training sample information, and a second to-be-processed file in the CSR data format is generated based on the combined training sample information. Since each combined training sample information contains the feature dimension number, the feature dimension number and the feature information represented by the feature dimension number can be directly read to realize the input / output operation of the training sample information, which greatly improves the efficiency of information processing.

[0123] In step 204, the server reads the number of feature dimensions of each training information in the second file to be processed, determines the amount of data to be read according to the number of feature dimensions, reads feature information corresponding to the number of feature dimensions based on the read data amount, and stores each training sample information in the first preset memory device.

[0124] Please continue to read Figure 4b The server can parse 22 the number of feature dimensions of each sample information to be trained in the second to-be-processed file through multiple threads (for example, 32 threads). For example, the number of feature dimensions can be 23. Since the number of feature dimensions can represent the subsequent entire feature information, the number of feature dimensions can be used to determine that the amount of data to be read subsequently is 23 data. Based on the read data amount, the feature information corresponding to the number of feature dimensions can be directly read, thereby realizing that the entire training sample information is stored in the first preset memory device with only two reads.

[0125] In step 205, the server performs randomization processing on each training sample information in the first preset memory device.

[0126] Please also refer to Figure 4b and Figure 4e , Figure 4e Another scenario diagram of the information processing method provided in the embodiment of the present application, the first preset memory device can be set to a shuffle memory, the shuffle memory can implement the shuffle method, that is, implement data scrambling 23, the shuffle memory can include 10 memory blocks, each memory block can store 100 batches of samples, the size of the batch can be arbitrarily set, for example, it can also be 50, that is, each memory block in the shuffle memory and the training sample information stored in the memory block are randomly shuffled. In the related art, positive samples and negative samples are continuous, which will cause the trained deep learning model to be not robust and the recognition accuracy will be relatively low. In this way, the embodiment of the present application scrambles the distribution of positive samples and negative samples in the first preset memory device through the shuffle method, such as Figure 4d As shown, the memory blocks of 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 can be shuffled into the order of memory blocks of 1, 2, 0, 4, 7, 9, 6, 3, 8, 5, and the training sample information of 100 batches stored in each memory block can also be shuffled to achieve uniform distribution of positive and negative samples.

[0127] Furthermore, when the order of the memory blocks is 1, 2, 0, 4, 7, 9, 6, 3, 8, 5, the global variable index pointer is still traversed in sequence, that is, the memory blocks 1, 2, 0, 4, 7, 9, 6, 3, 8, 5 and the scrambled training sample information in each memory block are read in sequence according to the scrambled order, that is, when index = 1, memory block 1 can be accessed, when index is 2, memory block 2 can be accessed, and so on, which will not be described in detail here.

[0128] Since the embodiment of the present application is multi-threaded data analysis, that is, in the memory block at the index position, there may be multiple threads simultaneously occupying the memory block at that location. In order to implement data protection, a lock mechanism may be added, that is, the global variable index may be used to determine which memory block is currently reading or storing the training sample information, and the local variable last_index may be assigned to each thread through the global variable index, where index=last-_index, and the memory block corresponding to the index is read or stored in the order of the time when the thread receives the local variable last_index, thereby implementing the addition of a lock mechanism to a specific memory block. After the training sample information in the memory block corresponding to the index is read out and stored, the global variable index is automatically incremented by 1 to implement unlocking. The reference code may be:

[0129]

[0130] In one implementation, when the index is equal to 10, 20, or a multiple of 10, the shuffle method is automatically executed to repeatedly shuffle the data 23 to ensure that the positive and negative samples are continuously evenly distributed.

[0131] In step 206, the server selects a first identifier from the first queue of the second preset memory device, determines the first target memory block corresponding to the first identifier, selects training sample information in sequence from the first preset memory device after randomization and stores it in the first target memory block, and when it is detected that there is no remaining storage space in the first target memory block, the first identifier is placed from the first queue to the second queue.

[0132] Please continue to read Figure 4b As shown, a second preset memory device 24 can be set as an intermediate cache state, and the second preset memory device can include 6 memory blocks, namely memory block 0, memory block 1 to memory block 5, and the numbers 0 to 5 are IDs (i.e., identifiers) corresponding to the memory blocks.

[0133] It should be noted that the embodiment of the present application can be composed of 34 threads, one thread for single-thread file name scanning, 32 threads for data parsing 22, and one thread for GPU collect (collect) data, that is, to realize the function of GPU data collector 25. Figure 2c As shown, the training sample information is read from the corresponding memory block according to the ID sequence of the memory queue 15. If the storage speed of a certain memory block is relatively slow, the single thread will wait for the storage of the memory block to be completed before reading it, which will cause congestion and reduce the information processing speed.

[0134] Therefore, in the embodiment of the present application, the second preset memory device 24 may include a first queue 241 and a second queue 242. The first queue 241 may be an empty queue, and the first queue 241 may store the IDs corresponding to the memory blocks that currently do not have any training sample information. The second queue 242 may be a full queue, and the second queue 242 may store the IDs corresponding to the memory blocks that are currently full, i.e., have no remaining storage space.

[0135] In this regard, please combine Figure 4f For reference, the server can select a first identifier (i.e., a memory block id) from the first queue 241 of the second preset memory device 24. The first identifier can be 0 or 2, indicating that no training sample information is currently stored in the memory block 0 or the memory block 2. Based on the first preset memory device after randomization, the training sample information is selected in turn and stored in the memory block 0 or the memory block 2 (i.e., the data is written into the memory block with the corresponding id).

[0136] Furthermore, when it is detected that there is no remaining space in the first target memory block, it means that the batch in the first target memory block has been filled, that is, no more training sample information can be stored, and the first identifier needs to be changed from the first queue 241 to the second queue 242 (that is, the memory block id is placed in the full queue, and mask [id] is set to true).

[0137] In step 207, the server selects a second identifier from the second queue of the second preset memory device, determines a second target memory block corresponding to the second identifier, and transfers the training sample information from the second target memory block to the image processor for processing. When it is detected that there is no training sample information in the second target memory block, the second identifier is placed from the second queue to the first queue.

[0138] Please continue to read Figure 4b and Figure 4fAs shown, the server can select a second identifier from the second queue 242 of the second preset memory device 24 (that is, take out the memory block id from the full queue, and set mask [id] to false). The second identifier can be 1, 5, 3, and 4, indicating that there is no remaining storage space in the corresponding second target memory block 242, that is, there is no remaining storage space in 1 memory block, 5 memory blocks, 3 memory blocks, or 4 memory blocks, and the training sample information in any of the memory blocks 1, 5 memory blocks, 3 memory blocks, or 4 memory blocks can be transferred to the image processor (that is, the data is read out from the memory block with the corresponding id).

[0139] Furthermore, when it is detected that there is no training sample information in the second target memory block, that is, the second target memory block is empty, it means that the batch in the second target memory block has been cleared, and the second identifier needs to be changed from the second queue 242 to the second queue 241 (that is, the memory block id is put into the empty queue). In this way, the thread that collects data for the GPU can find the memory block with a full storage status at any time according to the second identifier stored in the second queue 242 to read the training sample information to the GPU, thereby realizing data collection of the GPU data collector 25, so that the thread will not be blocked, further improving the efficiency of information processing.

[0140] Therefore, by supporting the tfrecord format, feature engineering can be adapted (seamlessly adaptable), and files in the tfrecord format can be improved to files in the csr format, reducing the frequent reading of io data, thereby improving the utilization of the cpu and gpu. Secondly, the design of the memory block is improved to support more threads to parse data concurrently, occupying less memory, and preventing the problem of a single thread parsing stuck and causing the entire process to be stuck. And by adding the shuffle function, the data distribution can be made more even, so that the training accuracy is higher.

[0141] As can be seen from the above, the embodiment of the present application obtains a first file to be processed in a first preset format; parses the first file to be processed, and converts the first file to be processed into a second file to be processed in a second preset format; reads the number of feature dimensions and corresponding feature information of each training sample information in the second file to be processed, and stores each training sample information in a first preset memory device; reads each training sample information in the first preset memory device into a graphics processor for processing. In this way, by adding the number of feature dimensions to the training sample information, the corresponding feature information can be quickly determined through the feature dimension data, and only two reading operations are required to store each training sample information in the first preset memory device, and then transferred to the graphics processor for processing. Compared with the prior art that requires frequent information reading operations, the embodiment of the present application can reduce information calculation and reading, greatly improving the efficiency of information processing.

[0142] Furthermore, since the empty memory blocks and the fully loaded memory blocks are marked by the first queue and the second queue, the threads can be processed quickly according to the storage status to avoid blocking, thereby better improving the efficiency of information processing.

[0143] In order to better implement the information processing method provided in the embodiment of the present application, the embodiment of the present application also provides a device based on the above information processing method. The meanings of the terms are the same as those in the above information processing method, and the specific implementation details can refer to the description in the method embodiment.

[0144] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of an information processing device provided in an embodiment of the present application, wherein the information processing device may include an acquisition unit 301, a parsing unit 302, a storage unit 303, and a reading unit 304, etc.

[0145] The acquisition unit 301 is used to acquire a first to-be-processed file in a first preset format.

[0146] In some embodiments, the acquisition unit 301 is used to:

[0147] Scanning the storage device based on a preset storage path;

[0148] The scanned file is processed in a first preset format to obtain a first to-be-processed file in the first preset format.

[0149] In some embodiments, the acquisition unit 302 is further configured to:

[0150] The first file to be processed in the first preset format is downloaded through the distributed file system.

[0151] The parsing unit 302 is used to parse the first file to be processed and convert the first file to be processed into a second file to be processed in a second preset format. The second file to be processed contains multiple training sample information, and the training sample information consists of the number of feature dimensions generated based on feature dimension statistics and corresponding feature information.

[0152] In some embodiments, the parsing unit 302 includes:

[0153] A parsing subunit, used for parsing the first file to be processed to obtain a corresponding feature name;

[0154] An acquisition subunit, configured to acquire the plain text information corresponding to the feature name when it is detected that the feature name matches the preset feature name stored in the preset feature table;

[0155] The conversion subunit is used to perform hash conversion on the plaintext information to obtain a corresponding hash key value;

[0156] A generating subunit is used to generate a second to-be-processed file in a second preset format according to the training sample information containing the hash key value.

[0157] In some embodiments, the generating subunit is used to:

[0158] Count the number of feature dimensions of each training sample information containing a hash key value, where the number of feature dimensions includes feature identifiers and feature value numbers;

[0159] Combining feature information of each training sample information including a hash key value and the corresponding number of feature dimensions to obtain combined training sample information;

[0160] A second to-be-processed file in a second preset format is generated based on the combined training sample information.

[0161] The storage unit 303 is used to read the number of feature dimensions and corresponding feature information of each training sample information in the second to-be-processed file, and store each training sample information in the first preset memory device.

[0162] In some embodiments, the storage unit 303 is used to:

[0163] Read the number of feature dimensions of each training sample information in the second to-be-processed file;

[0164] Determine the amount of data to be read according to the number of feature dimensions;

[0165] The feature information corresponding to the number of feature dimensions is read based on the read data amount, and each training sample information is stored in a first preset memory device.

[0166] The reading unit 304 is used to read each training sample information in the first preset memory device into the graphics processor for processing.

[0167] In some embodiments, the reading unit 304 includes:

[0168] A random subunit, used for performing random processing on each training sample information in the first preset memory device;

[0169] A storage subunit, used for sequentially selecting training sample information from the first preset memory device after randomization and storing it in a second preset memory device;

[0170] The reading subunit is used to read each training sample information in the second preset memory device into the graphics processor for processing.

[0171] In some embodiments, the second preset memory device includes a first queue and a second queue, and the storage subunit is used to:

[0172] Selecting a first identifier from a first queue in a second preset memory device;

[0173] Determine a first target memory block corresponding to the first identifier;

[0174] Sequentially selecting training sample information from the first preset memory device after randomization and storing it in the first target memory block;

[0175] When it is detected that there is no remaining storage space in the first target memory block, the first identifier is put from the first queue to the second queue.

[0176] In some embodiments, the reading subunit is used to:

[0177] Selecting a second identifier from a second queue of the second preset memory device;

[0178] Determine a second target memory block corresponding to the second identifier;

[0179] Transferring the training sample information in the second target memory block to an image processor for processing;

[0180] When it is detected that the second target memory block does not contain the training sample information, the second identifier is put from the second queue into the first queue.

[0181] The specific implementation of each of the above units can be found in the previous embodiments, which will not be described in detail here.

[0182] As can be seen from the above, the embodiment of the present application obtains the first to-be-processed file of the first preset format through the acquisition unit 301; the parsing unit 302 parses the first to-be-processed file and converts the first to-be-processed file into the second to-be-processed file of the second preset format; the storage unit 303 reads the number of feature dimensions and the corresponding feature information of each training sample information in the second to-be-processed file, and stores each training sample information in the first preset memory device; the reading unit 304 reads each training sample information in the first preset memory device into the graphics processor for processing. In this way, by adding the number of feature dimensions in the training sample information, the corresponding feature information can be quickly determined through the feature dimension data, and only two reading operations are required to store each training sample information in the first preset memory device, and then transferred to the graphics processor for processing. Compared with the prior art that requires frequent information reading operations, the embodiment of the present application can reduce information calculation and reading, greatly improving the efficiency of information processing.

[0183] The present application also provides a server, such as Figure 6As shown, it shows a schematic diagram of the structure of the server involved in the embodiment of the present application, specifically:

[0184] The server may include one or more processing core processors 401, one or more computer-readable storage media memories 402, a power supply 403, an input unit 404 and other components. Those skilled in the art will appreciate that Figure 6 The server structure shown in the figure does not constitute a limitation on the server, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0185] The processor 401 is the control center of the server, and uses various interfaces and lines to connect various parts of the entire server. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, the processor 401 performs various functions of the server and processes data, thereby monitoring the server as a whole. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0186] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the server, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0187] The server also includes a power supply 403 for supplying power to each component. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions. The power supply 403 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0188] The server may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0189] Although not shown, the server may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the server will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby realizing various functions, as follows:

[0190] Acquire a first file to be processed in a first preset format; parse the first file to be processed and convert the first file to be processed into a second file to be processed in a second preset format, wherein the second file to be processed includes a plurality of training sample information, and the training sample information consists of the number of feature dimensions and corresponding feature information generated based on feature dimension statistics; read the number of feature dimensions and the corresponding feature information of each training sample information in the second file to be processed, and store each training sample information in a first preset memory device; read each training sample information in the first preset memory device into a graphics processor for processing.

[0191] In the above embodiments, the description of each embodiment has its own focus. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the information processing method above, and will not be repeated here.

[0192] As can be seen from the above, the server of the embodiment of the present application can obtain a first file to be processed in a first preset format; parse the first file to be processed, and convert the first file to be processed into a second file to be processed in a second preset format; read the number of feature dimensions and corresponding feature information of each training sample information in the second file to be processed, and store each training sample information in a first preset memory device; read each training sample information in the first preset memory device into a graphics processor for processing. In this way, by adding the number of feature dimensions to the training sample information, the corresponding feature information can be quickly determined through the feature dimension data, and only two reading operations are required to store each training sample information in the first preset memory device, and then transfer it to the graphics processor for processing. Compared with the prior art that requires frequent information reading operations, the embodiment of the present application can reduce information calculation and reading, greatly improving the efficiency of information processing.

[0193] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0194] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any one of the information processing methods provided in the embodiments of the present application. For example, the instructions can execute the following steps:

[0195] Acquire a first file to be processed in a first preset format; parse the first file to be processed and convert the first file to be processed into a second file to be processed in a second preset format, wherein the second file to be processed includes a plurality of training sample information, and the training sample information consists of the number of feature dimensions and corresponding feature information generated based on feature dimension statistics; read the number of feature dimensions and the corresponding feature information of each training sample information in the second file to be processed, and store each training sample information in a first preset memory device; read each training sample information in the first preset memory device into a graphics processor for processing.

[0196] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0197] Since the instructions stored in the computer-readable storage medium can execute the steps in any information processing method provided in the embodiments of the present application, the beneficial effects that can be achieved by any information processing method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0198] According to one aspect of an embodiment of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations provided in the above embodiments.

[0199] The above is a detailed introduction to an information processing method, device and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An information processing method, characterized in that: include: Acquire a first file to be processed in a first preset format; Parsing the first to-be-processed file, converting the first to-be-processed file into a second to-be-processed file in a second preset format, wherein the second to-be-processed file includes a plurality of training sample information, wherein the training sample information includes a number of feature dimensions generated based on feature dimension statistics and corresponding feature information; Reading the number of feature dimensions and corresponding feature information of each training sample information in the second to-be-processed file, and storing each training sample information in a first preset memory device; Reading each training sample information in the first preset memory device into a graphics processor for processing; The step of reading the number of feature dimensions and corresponding feature information of each training sample information in the second to-be-processed file, and storing each training sample information in a first preset memory device, includes: Read the number of feature dimensions of each training sample information in the second to-be-processed file; Determining the amount of data to be read according to the number of characteristic dimensions; The feature information corresponding to the number of feature dimensions is read based on the read data amount, and each training sample information is stored in a first preset memory device.

2. The information processing method according to claim 1, characterized in that: The parsing the first to-be-processed file and converting the first to-be-processed file into a second to-be-processed file in a second preset format includes: Parsing the first to-be-processed file to obtain corresponding feature names; When it is detected that the feature name matches a preset feature name stored in the preset feature table, obtaining plain text information corresponding to the feature name; Performing hash conversion on the plaintext information to obtain a corresponding hash key value; A second to-be-processed file in a second preset format is generated according to the training sample information including the hash key value.

3. The information processing method according to claim 2, characterized in that: The step of generating a second to-be-processed file in a second preset format according to the training sample information including the hash key value includes: Counting the number of feature dimensions of each training sample information including a hash key value, wherein the number of feature dimensions includes a feature identifier and a number of feature values; Combining feature information of each training sample information including a hash key value and the corresponding number of feature dimensions to obtain combined training sample information; A second to-be-processed file in a second preset format is generated based on the combined training sample information.

4. The information processing method according to any one of claims 1 to 3, characterized in that: The step of reading each training sample information in the first preset memory device into a graphics processor for processing includes: Randomizing each training sample information in the first preset memory device; Sequentially select training sample information from the first preset memory device after randomization and store it in the second preset memory device; Each training sample information in the second preset memory device is read into the graphics processor for processing.

5. The information processing method according to claim 4, characterized in that: The second preset memory device includes a first queue and a second queue, and the step of sequentially selecting training sample information from the first preset memory device after randomization and storing it in the second preset memory device includes: Selecting a first identifier from a first queue in a second preset memory device; Determine a first target memory block corresponding to the first identifier; Sequentially selecting training sample information from the first preset memory device after randomization and storing it in the first target memory block; When it is detected that there is no remaining storage space in the first target memory block, the first identifier is placed from the first queue to the second queue.

6. The information processing method according to claim 5, characterized in that: The step of reading each training sample information in the second preset memory device into a graphics processor for processing includes: Selecting a second identifier from a second queue of the second preset memory device; Determine a second target memory block corresponding to the second identifier; Transferring the training sample information in the second target memory block to an image processor for processing; When it is detected that no training sample information exists in the second target memory block, the second identifier is put from the second queue into the first queue.

7. The information processing method according to any one of claims 1 to 3, characterized in that: The step of obtaining a first file to be processed in a first preset format includes: Scanning the storage device based on a preset storage path; The scanned file is processed in a first preset format to obtain a first to-be-processed file in the first preset format.

8. The information processing method according to any one of claims 1 to 3, characterized in that: The step of obtaining a first file to be processed in a first preset format includes: The first file to be processed in the first preset format is downloaded through a distributed file system.

9. An information processing device, characterized in that: include: An acquiring unit, configured to acquire a first file to be processed in a first preset format; a parsing unit, configured to parse the first to-be-processed file and convert the first to-be-processed file into a second to-be-processed file in a second preset format, wherein the second to-be-processed file includes a plurality of training sample information, wherein the training sample information includes a number of feature dimensions generated based on feature dimension statistics and corresponding feature information; A storage unit, used for reading the number of feature dimensions and corresponding feature information of each training sample information in the second to-be-processed file, and storing each training sample information in a first preset memory device; A reading unit, used for reading each training sample information in the first preset memory device into a graphics processor for processing; Wherein, the storage unit is used for: Read the number of feature dimensions of each training sample information in the second to-be-processed file; Determining the amount of data to be read according to the number of characteristic dimensions; The feature information corresponding to the number of feature dimensions is read based on the read data amount, and each training sample information is stored in a first preset memory device.

10. The information processing device according to claim 9, characterized in that: The parsing unit comprises: A parsing subunit, used for parsing the first to-be-processed file to obtain a corresponding feature name; An acquisition subunit, configured to acquire plain text information corresponding to the feature name when it is detected that the feature name matches a preset feature name stored in a preset feature table; The conversion subunit is used to perform hash conversion on the plaintext information to obtain a corresponding hash key value; A generating subunit is used to generate a second to-be-processed file in a second preset format according to the training sample information containing the hash key value.

11. The information processing device according to claim 10, characterized in that: The generating subunit is used for: Counting the number of feature dimensions of each training sample information including a hash key value, wherein the number of feature dimensions includes a feature identifier and a number of feature values; Combining feature information of each training sample information including a hash key value and the corresponding number of feature dimensions to obtain combined training sample information; A second to-be-processed file in a second preset format is generated based on the combined training sample information.

12. The information processing device according to any one of claims 9 to 11, characterized in that: The reading unit comprises: A random subunit, used for performing random processing on each training sample information in the first preset memory device; A storage subunit, used for sequentially selecting training sample information from the first preset memory device after randomization and storing it in a second preset memory device; The reading subunit is used to read each training sample information in the second preset memory device into the graphics processor for processing.

13. The information processing device according to claim 12, characterized in that: The second preset memory device includes a first queue and a second queue, and the storage subunit is used for: Selecting a first identifier from a first queue in a second preset memory device; Determine a first target memory block corresponding to the first identifier; Sequentially selecting training sample information from the first preset memory device after randomization and storing it in the first target memory block; When it is detected that there is no remaining storage space in the first target memory block, the first identifier is placed from the first queue to the second queue.

14. The information processing device according to claim 13, characterized in that: The reading subunit is used for: Selecting a second identifier from a second queue of the second preset memory device; Determine a second target memory block corresponding to the second identifier; Transferring the training sample information in the second target memory block to an image processor for processing; When it is detected that no training sample information exists in the second target memory block, the second identifier is put from the second queue into the first queue.

15. The information processing device according to any one of claims 9 to 11, characterized in that: The acquisition unit is used to: Scanning the storage device based on a preset storage path; The scanned file is processed in a first preset format to obtain a first to-be-processed file in the first preset format.

16. The information processing device according to any one of claims 9 to 11, characterized in that: The acquisition unit is further used for: The first file to be processed in the first preset format is downloaded through a distributed file system.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the information processing method according to any one of claims 1 to 8.

18. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the information processing method according to any one of claims 1 to 8 are implemented.

19. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the information processing method according to any one of claims 1 to 8 when executing the computer program.

Images