Data processing method, prediction method, device, storage medium, and program product

Abstract

The application provides a data processing method, a prediction method, equipment, a storage medium and a program product, wherein the method comprises: determining a current batch index to be processed; wherein the batch index to be processed changes in sequence during model training; if it is determined according to the batch index that a round of training has been completed based on a current file list, the file identifiers in the file list are randomly sorted; wherein the file list comprises file identifiers corresponding to a plurality of files in a disk, and the files comprise training data of at least one batch; according to the sequence of the file identifiers in the file list, a target file to which the training data corresponding to the batch index belongs is determined; if the target file is inconsistent with a file in the memory, the target file in the disk is loaded into the memory, and the training data corresponding to the batch index is read from the target file to perform model training. The application can save the time for reading the training data, improve the data reading speed, and improve the model training efficiency.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data processing method, prediction method, device, storage medium, and program product. Background Technology

[0002] With the continuous development of artificial intelligence and big data technologies, the size of training data used by artificial intelligence models is also constantly increasing.

[0003] In some model training processes, hundreds of gigabytes or even more of training data are used. Due to their large size, they cannot be loaded into memory all at once. Therefore, data may need to be read from the disk during training. In order to optimize the model parameters, data often needs to be read randomly, which results in a lot of disk addressing time. The performance is usually low when reading data, which leads to slow model training and low overall training efficiency. Summary of the Invention

[0004] The main objective of this application is to provide a data processing method, prediction method, device, storage medium, and program product to improve the efficiency of model training.

[0005] In a first aspect, embodiments of this application provide a data processing method, including:

[0006] Determine the current batch index to be processed; the order of the batch indices to be processed changes during model training.

[0007] If it is determined that a training round has been completed based on the current file list, the file identifiers in the file list are randomly sorted; wherein, the file list includes file identifiers corresponding to multiple files on the disk, and the files contain at least one batch of training data;

[0008] Based on the order of file identifiers in the file list, determine the target file to which the training data corresponding to the batch index belongs;

[0009] If the target file is inconsistent with the file in memory, the target file in the disk is loaded into memory, and the training data corresponding to the batch index is read from the target file for model training.

[0010] Optionally, the target file to which the training data corresponding to the batch index belongs is determined according to the order of file identifiers in the file list, including:

[0011] Based on the number of batches contained in a single file and the batch index, determine the sequence number of the file identifier of the target file in the file list;

[0012] The corresponding target file is determined based on the order of the file identifiers in the file list and the sequence number;

[0013] The number of batches contained in a single file is determined by a pre-set batch size and the number of training data sets contained in a single file; the batch size is used to represent the number of training data sets contained in a single batch.

[0014] Optionally, the method further includes:

[0015] If the target file is the same as the file in memory, then the position of the training data corresponding to the batch index in the target file is calculated based on the batch index, batch size, and number of batches contained in a single file;

[0016] The training data corresponding to the batch index is read from the corresponding position of the target file in memory, and the model is trained based on the read training data.

[0017] Optionally, the method further includes:

[0018] Calculate the range of values ​​for the batch index based on the number of training data sets contained in the multiple files and the batch size;

[0019] After training the model based on the training data corresponding to the batch index, the batch index to be processed is updated to the current batch index plus one. Alternatively, if the current batch index is the maximum value in the range of values, the updated batch index is set to the minimum value in the range of values, so as to continue processing the updated batch index.

[0020] Accordingly, if it is determined from the batch index that a training round has been completed based on the current file list, then the file identifiers in the file list are randomly sorted, including:

[0021] If the batch index is the minimum value in the range, it is determined that one round of training has been completed based on the current file list, and the file identifiers in the file list are randomly sorted.

[0022] Optionally, the target file on the disk is loaded into memory, and the training data corresponding to the batch index is read from the target file for model training, including:

[0023] The target file in the disk is loaded into memory, and the training data contained in the target file is randomly sorted.

[0024] Based on the batch index, the corresponding training data is read from the randomly sorted target file for model training.

[0025] Optionally, the model is used to predict the probability that a user selects a certain product model; the training data includes user feature information, product model feature information, and labels, the labels being used to indicate whether the user selects the product model; before training the model, the following is also included:

[0026] The training dataset used to train the model is downloaded to the disk, and the training dataset is divided into the multiple files for storage;

[0027] The training dataset is presented to personnel involved in model training so that they can process abnormal data in the training dataset; and / or, abnormal training data in the training dataset is detected and processed based on the median of each feature information in the training dataset.

[0028] Secondly, embodiments of this application also provide a prediction method, including:

[0029] Based on the search information entered by the user, at least one corresponding product model is found from the product model library;

[0030] Based on the user's information and the information of at least one product model, the selection probability of each product model is determined based on the search recommendation model, and the product models are ranked according to the selection probability.

[0031] Display at least one product model to the user based on the sorting results;

[0032] The search recommendation model is a model trained according to any one of the methods described in the first aspect.

[0033] Thirdly, embodiments of this application provide an electronic device, including:

[0034] At least one processor; and

[0035] A memory that is communicatively connected to the at least one processor;

[0036] The memory stores instructions that can be executed by the at least one processor to cause the electronic device to perform the method described in any of the above aspects.

[0037] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method described in any of the above aspects.

[0038] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the methods described in any of the above aspects.

[0039] The data processing method, prediction method, device, storage medium, and program product provided in this application embodiment can determine the current batch index to be processed. During model training, the order of the batch indexes to be processed changes. If it is determined that one round of training has been completed based on the current file list, the file identifiers in the file list are randomly sorted. The file list includes file identifiers corresponding to multiple files on the disk, and each file contains at least one batch of training data. Based on the order of the file identifiers in the file list, the target file to which the training data corresponding to the batch index belongs is determined. If the target file is inconsistent with the file in memory, the target file on the disk is loaded into memory, and the training data corresponding to the batch index is read from the target file for model training. This allows for searching for the corresponding file based on the batch index and comparing it with the file in memory to determine whether to directly read the training data from memory or load the file from the disk into memory before reading the training data. The changing batch index order can likely hit the file in memory, saving time in reading training data while maintaining a certain degree of randomness, improving data reading speed, and enhancing model training efficiency. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0041] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0042] Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0043] Figure 3 A schematic diagram illustrating a batch and round arrangement provided for an embodiment of this application;

[0044] Figure 4 A schematic diagram illustrating a data storage process provided in an embodiment of this application;

[0045] Figure 5 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0046] Figure 6 This application provides a schematic diagram illustrating the use of a search recommendation model in an embodiment of the present application.

[0047] Figure 7A flowchart illustrating a prediction method provided in an embodiment of this application;

[0048] Figure 8 A schematic diagram illustrating the use of a home decoration scene model provided in an embodiment of this application;

[0049] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0050] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0051] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0052] First, let me explain the terms used in this application:

[0053] Disk caching: loading data from the disk into memory in advance using certain methods; the disk can also be called a hard drive, and memory can also be called a cache.

[0054] Random read: When there are multiple sets of training data, the training data is read randomly, which is different from sequential read.

[0055] Model training epochs and batches: When training a model based on a training dataset, it can go through multiple epochs, each epoch can contain multiple batches, and each batch can contain multiple training data points. Each time the model parameters are updated, they can be updated based on the data from one batch. After all the data has been updated in one epoch, it represents the completion of one epoch, and then the next epoch begins. The order of the training data in the next epoch can be different from that in the current epoch.

[0056] Batch index: Used to identify each batch. For example, index 1 can point to the training data of the first batch.

[0057] The embodiments of this application can be applied to model training schemes in any field. Generally, the model can be trained based on a training dataset. The training dataset may include multiple training data sets. By inputting the training data into the model, corresponding prediction results can be obtained. Then, the corresponding loss value is determined based on the prediction results. The model parameters are adjusted based on the loss value using methods such as gradient descent until the number of training epochs or the model's accuracy meets the preset requirements, at which point training ends.

[0058] To improve model training performance, random reads are typically used to access training data. Furthermore, the training dataset can contain tens of millions or even more data points. Due to memory limitations, it's usually difficult to pre-load all the data into memory. Therefore, during model training, it's inevitable to read data from disk, which is inefficient. This application aims to solve the problem of high-performance reading of large datasets stored on disk, thereby improving training speed.

[0059] In some techniques, a mini-batch reading solution based on pandas can be used. Specifically, pandas provides the tools needed to efficiently manipulate large datasets. When it is necessary to read training datasets from disk, the pandas library can be used, specifying the skiprows and nrows parameters to achieve the purpose of reading small batches of data from disk.

[0060] In this scheme, the batch index used during model training varies randomly, for example, batch 1, batch 1000, batch 20, and so on. When reading data, the corresponding batch of training data needs to be addressed sequentially from the disk.

[0061] This approach has two drawbacks. First, each time training data is read, it is necessary to locate the corresponding position on the disk. Multiple training data sets require multiple addressing operations, which results in a large amount of disk addressing time when the data volume is large, leading to low efficiency. Second, it can only read nrows of data at a time, and cannot perform global or near-global random shuffle (out-of-order) of the data, resulting in poor model training performance.

[0062] In other technologies, the TensorFlow framework can be used as a tool for model training, with data reading based on tfRecord. tfRecord is a data format provided by TensorFlow, but it has two drawbacks. First, many training datasets do not support downloading data in tfRecord format, requiring additional code to perform save and read operations. Second, tfRecord is a simple format for storing binary record sequences after serialization. If problems are found in some data during data processing or training, it cannot be viewed or modified, which can significantly affect the efficiency of model training and tuning.

[0063] In view of this, embodiments of this application provide a data processing method based on disk buffering, which can change the batch index used by the model from unordered to ordered, preload files in random order and randomly sort the data in the files, and then put them into memory. Through a certain algorithm, the batch index currently used is made to hit the data in memory with a high probability, and the randomness of the training data is guaranteed throughout the process, thereby accelerating the training of the model.

[0064] Figure 1 This is a schematic diagram illustrating an application scenario provided by an embodiment of this application. For example... Figure 1 As shown, before training the model, the training dataset can be downloaded from the data platform to the local disk. The training dataset is distributed and stored in multiple files on the local disk. After each round of training, the file list can be shuffled.

[0065] For example, the training dataset contains 30,000 training data points, stored in files A, B, and C respectively, with each file containing 10,000 training data points. After each round of training ends and before the next round of training begins, the order of the file list can be shuffled. For example, during the first round of training, the file order is A, B, C. Before starting the second round of training, the file list is randomly reordered to file B, C, A, and the second round of training is performed according to the order of file B, C, A.

[0066] During training, batch indices can be set to change sequentially, such as batch 1, batch 2, batch 3, and so on. After randomly sorting the file list, the current batch index can be used to calculate which file it matches based on the training data contained in each file. For example, if each batch contains 10 training data sets, then each file corresponds to 1000 batches. When the current batch index is 1001, it means that the corresponding batch is in the second file of the sorted list.

[0067] If the second file in the sorted list is file C, then we can check if file C exists in memory. If it does, we retrieve it directly from memory.

[0068] The training data is read from file C. If it does not exist, file C is loaded from disk into memory and the corresponding training data is read for model training.

[0069] Optionally, when loading file C into memory, the training data in file C can be randomly arranged, shuffling the original order of the training data in the file, and then the training data to be used can be retrieved from it according to the batch index.

[0070] The training data read in the current batch is used to update the model parameters. After reading the training data in the current batch, the next batch of training data can be read in a similar way until all the data in files B, C, and A have been read once. The current round ends, and the file list can be randomly sorted again before starting the next round of training.

[0071] During each data read, since the batch index changes sequentially, there is an implicit sequential correspondence between the batch index and the training data contained in the file. Therefore, the batch index will likely hit the file in memory. For example, if each file contains 1000 batches of training data, then in 1000 reads, there will only be 1 chance that the file does not exist in memory and needs to be loaded from the disk. In most other cases, the data can be read directly from memory, saving the time of loading from the disk to memory.

[0072] Compared to the aforementioned pandas-based mini-batch reading scheme, this embodiment can make the batch index ordered, thereby sequentially reading the training data corresponding to the randomly sorted file list from memory, reducing disk access time. In addition, pandas reading is limited to the range of skiprows to skiprows+nrows, which lacks randomness. This embodiment splits the large dataset into multiple small files that can be stored in memory, randomly sorts the file list, and can also randomly sort the training data within the files when the files are loaded into memory, thus improving the randomness of the data.

[0073] Compared to the batch data reading scheme of tfRecord mentioned above, the embodiments of this application do not require writing additional code for storing and loading tfRecord. They can directly use formats such as CSV supported by conventional data platforms to store data, eliminating the need for additional code. In addition, when there is abnormal data in the training data, because tfRecord is stored as a binary stream, it is impossible to view and remove the abnormal data. The embodiments of this application do not require binary stream conversion and can use editable and interpretable conventional formats to store data, reducing the problem of data anomalies.

[0074] In summary, the data processing method provided in this application can not only achieve global near-random reading of large datasets, but also achieve near-high-speed memory storage and data reading speed through reasonable algorithms of batch indexing and pre-loaded files. This can accelerate the training speed of models when large datasets are stored on disk and solve the problem of long loading time from disk for training data.

[0075] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Where there is no conflict between the embodiments, the following embodiments and features can be combined with each other. Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0076] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application. The execution entity of the method in this embodiment can be applied to any device or module with data processing capabilities. For example... Figure 2 As shown, the method may include:

[0077] Step 201: Determine the index of the batch to be processed.

[0078] During model training, the order of the batch indices to be processed changes. The model can be a neural network model.

[0079] Optionally, the batch index to be processed can be the batch index corresponding to the training data to be used during model training. Each batch index can correspond to a batch of training data, and a batch can contain at least one training data piece.

[0080] During model training, as the training progresses, a continuous stream of training data is fed into the model to train it. The batch indices to be processed can change sequentially, such as increasing or decreasing sequentially, rather than changing randomly. Examples of sequentially increasing indices: batch 0, batch 1, batch 2, ...; examples of sequentially decreasing indices: batch 9999, batch 9998, ... Examples of randomly changing indices: batch 1, batch 1001, batch 0, ...

[0081] Step 202: If it is determined that a training round has been completed based on the current file list, then the file identifiers in the file list are randomly sorted.

[0082] The file list includes file identifiers corresponding to multiple files on the disk, and each file contains at least one batch of training data.

[0083] Optionally, when processing each batch of training data, the index of the current batch to be processed can be determined first, and the training round can be judged based on the batch index.

[0084] Optionally, if the current batch index is detected as 0 when the batch index is increasing, it means that a new round has been entered, that is, a round of training has been completed based on the current file list, and at this time the file list can be randomly sorted.

[0085] Alternatively, if the batch index is decreasing in order, and the current batch index is detected as the maximum index, it means that one round of training has been completed based on the current file list, and the training data contained in all files in the current file list has been used once. After completing one round of training, the file identifiers in the file list can be randomly sorted.

[0086] In other alternative implementations, the model training software may also generate a corresponding trigger signal after the model has completed one training iteration to instruct the file list to be randomly sorted. This embodiment does not limit the execution order of the steps; for example, step 202 may be executed before step 201, or simultaneously with step 201.

[0087] It should be noted that during model training, the order of files stored on the disk can remain unchanged. Each file can have a corresponding file identifier, such as a filename or a file index. After each training round, only the order of the file identifiers in the file list needs to be changed. The file list can be saved in a table or other ways.

[0088] Step 203: Determine the target file to which the training data corresponding to the batch index belongs based on the order of the file identifiers in the file list.

[0089] Optionally, after the file identifiers in the file list are randomly sorted, the batch index and the training data in the file can have a certain correspondence. For example, after random sorting, the order of the file identifiers is: file A, file B, file C. In each file, 10 training data sets form a batch. Then, index 0 can correspond to the first 10 training data sets in file A, index 1 can correspond to the 11th to 20th training data sets in file A, and so on.

[0090] It should be noted that the above correspondence is only an example. In practical applications, various algorithms can be used to determine the file corresponding to the batch index, as long as there is a certain mapping relationship between the batch index and the file.

[0091] In one alternative implementation, when processing the current batch index to be processed, the file corresponding to the batch index can be determined according to a preset algorithm.

[0092] In another alternative implementation, after the file list is randomly sorted, the training data corresponding to each batch of indices can be determined immediately. When processing each batch of indices to be processed, the file to which the corresponding training data belongs can be found directly.

[0093] Step 204: If the target file is inconsistent with the file in memory, load the target file from the disk into memory and read the training data corresponding to the batch index from the target file for model training.

[0094] After identifying the target file, it can be compared with the files in memory. Optionally, at least one file can be cached in memory. The target file not matching the files in memory specifically means that the target file is different from all the files in memory; that is, the target file has not yet been loaded into memory. In this case, the target file can be loaded into memory from disk.

[0095] If the target file is consistent with the file in memory, that is, the target file is already cached in memory, then the training data corresponding to the batch index can be directly read from the target file for model training.

[0096] Figure 3 This is a schematic diagram illustrating a batch and round arrangement as provided in an embodiment of this application. (For example...) Figure 3 As shown, the rectangles represent files, and the black bars within the rectangles represent training data. Each file can contain multiple sets of training data. After iterating through the training data in all the files once, one round of training is completed. Each round can contain multiple batches. After updating the model parameters based on each batch of training data, the model parameters are further updated based on the next batch of training data.

[0097] Optionally, after the training data for the current batch index has been read, the batch index can be incremented by 1, and steps 201 to 204 above can be repeated. After one training round is completed, the batch index can be reset, and the next training round can be performed again by referring to the above steps.

[0098] For example, the training dataset used to train a model contains about 100 million training data sets. Assuming that each file contains 128,000 training data sets, then approximately 100 million divided by 128,000 equals about 780 files. If each batch contains 128 training data sets, then each file can correspond to 1,000 batches.

[0099] In practical applications, at the start of model training, a file list can be generated based on the file identifiers corresponding to the files on the disk. When the batch index is equal to 0, the file ranked first in the file list is loaded into memory according to steps 201 to 204 above, and the first 128 training data of the file are read from memory for model training.

[0100] Next, the batch index is incremented by 1. With the batch index equal to 1, steps 201 to 204 above are executed. Since the file corresponding to batch index 1 already exists in memory, data entries 129 to 256 of that file can be read directly from memory. This process continues until the batch index equals 999. After executing steps 201 to 204, all 100 million training data entries have been used once, and the model has completed one round of training.

[0101] After a training round is completed, the file list can be randomly sorted. After random sorting, the next training round can be performed following the same process as described above. The training process for the next round is similar to the previous process, except that the order of the file identifiers in the file list has changed. This allows different training data to be used in different orders in different rounds, thus meeting the randomness requirement of model training, even as the batch index order increases.

[0102] After each round, the order of the file identifiers changes, which is equivalent to a change in the batch index for each file. When loading each file, the system checks if the corresponding file is in memory according to the currently needed batch index. If it is, it is read and used directly; otherwise, the file is loaded. Because the batch index changes sequentially, there will be 999 successful hits out of every 1000 checks, resulting in a very high hit rate.

[0103] In summary, the data processing method provided in this embodiment sets the batch index during model training to be sequentially variable. Furthermore, during training, multiple files containing training data are continuously and randomly sorted. The corresponding file is searched according to the batch index and compared with the file in memory to determine whether to read the training data directly from memory or load the file from disk into memory before reading the training data. The sequentially variable batch index can hit the file in memory with a high probability, which can save the time of reading training data, improve the data reading speed, and improve the model training efficiency while ensuring a certain degree of randomness.

[0104] Optionally, before starting to train the model, the training dataset used to train the model can be downloaded to the disk and the training dataset can be divided into the multiple files for storage. Figure 4 This is a schematic diagram illustrating a data storage process provided in an embodiment of this application. For example... Figure 4As shown, data can be stored in a data platform such as ODPS (Open Data Processing Service). After the start, a portion of the training data can be downloaded from ODPS. During the download, each training data item in ODPS can be located based on offset and limit. For example, when offset = m and limit = n, the training data items from m to m+n can be downloaded.

[0105] Furthermore, the filenames of the training data can be calculated based on the download batches. Here, each download batch can correspond to one file. For example, the filenames of the first 128,000 training data in ODPS are named file1, and the filenames of the 128,001 to 256,000 training data are named file2.

[0106] After determining the filename corresponding to the downloaded training data, you can check if the file with that filename already exists locally. If it doesn't exist, create the file with that filename to store the data and add the corresponding training data to that file. If the file with that filename already exists locally, you can use the append method to add the training data to the end of the file.

[0107] Optionally, multiple threads can be used to download the training data stored in ODPS to the local disk in batches. For example, after thread 1 downloads the first 60,000 training data and saves it to the created file 1, thread 2 can download the 60,001st to 128,000th training data and append it to the end of file 1.

[0108] Using the above method, the training dataset can be downloaded to the local disk and saved in multiple files. Optionally, these files can be CSV (Comma-Separated Values) files, which can store data in plain text format, and the delimiter characters in the files can be commas or other symbols. Alternatively, the files can be XLSX files or any other file system capable of storing data.

[0109] Optionally, each file may include at least one line. In this case, a set of training data can also be referred to as a single training data point, with each line representing a single training data point. The training data may include sample data and corresponding labels. For example, for an image classification model, the corresponding sample data may be pixel values ​​in an image, and the label may be the image category.

[0110] Optionally, before training the model, the training dataset can be shown to the personnel involved in model training so that the personnel can process the abnormal data in the training dataset; and / or, abnormal training data in the training dataset can be detected and processed based on the median of each feature information in the training dataset.

[0111] In one alternative implementation, the training dataset can be downloaded to a local disk and then displayed. Trainers can view the various training data in the training dataset and perform operations such as deleting or modifying abnormal training data.

[0112] In another alternative implementation, the device can automatically process the training dataset. Optionally, each training data point can include multiple feature information, and the median of each feature information can be used to determine whether each training data point is abnormal.

[0113] For example, the training dataset includes L training data points, each including h feature information items. For any feature information item, the median corresponding to that feature information item in the L training data points is calculated. For each training data point, if the difference or ratio between a certain feature information item and its corresponding median exceeds a preset range, the training data point is considered abnormal and can be deleted.

[0114] In summary, the data storage and exception handling processes described above enable the rapid preparation of multiple files to support model training, the processing of abnormal data within these files to improve the accuracy of model training, and the support for users to independently view and modify data, thereby enhancing the user experience.

[0115] In one or more embodiments of this application, optionally, determining the target file to which the training data corresponding to the batch index belongs based on the order of file identifiers in the file list includes: determining the sequence number of the file identifier of the target file in the file list based on the number of batches contained in a single file and the batch index; determining the corresponding target file based on the order of each file identifier in the file list and the sequence number; wherein, the number of batches contained in a single file is determined by a preset batch size and the number of training data sets contained in a single file; the batch size is used to represent the number of training data sets contained in a single batch.

[0116] Optionally, the batch size can be user-inputted, a default setting can be used, or the batch size can be automatically determined based on requirements such as the training dataset, computational performance, time, and accuracy. After determining the batch size, the number of training data sets in a single file can be divided by the batch size to obtain the number of batches in a single file. The index of the current batch to be processed can be divided by the number of batches in a single file, rounded down, and then incremented by 1 to obtain the sequence number of the target file.

[0117] Since the file identifiers in the file list are randomly sorted, the order of the file identifiers may not be consistent with the order of the files on the disk. When determining the target file, the corresponding file identifier can be found in the file list based on the calculated sequence number, and the file corresponding to the found file identifier can be used as the target file.

[0118] In summary, by calculating the number of batches in a single file based on the pre-set batch size and the number of training data copies contained in a single file, and then combining the number of batches in a single file with the index of the batches to be processed, along with a randomly sorted list of files, the target file can be quickly and accurately determined, improving the overall efficiency and accuracy of reading training data.

[0119] In one or more embodiments of this application, optionally, if the target file is consistent with the file in memory, the position of the training data corresponding to the batch index in the target file is calculated based on the batch index, batch size, and the number of batches contained in a single file; the training data corresponding to the batch index is read from the corresponding position of the target file in memory, and model training is performed based on the read training data.

[0120] Specifically, when the target file matches the file in memory, training data can be read directly from the target file in memory. The index of the current batch to be processed can be divided by the number of batches contained in a single file, the quotient can be rounded down and added 1 to obtain the index of the target file, and the remainder can be used to indicate which batch in the target file the batch index corresponds to.

[0121] Optionally, the remainder is multiplied by the batch size, and the product is incremented by 1 to obtain the first row number; alternatively, the remainder is incremented by 1 and multiplied by the batch size to obtain the second row number. In this embodiment, the position of the training data corresponding to the batch index in the target file is recorded as the target position, which is between the first row number and the second row number.

[0122] For example, if the remainder obtained by dividing the batch index by the number of batches contained in a single file is R, and the batch size is S, then the target position is from line R / S-1 to line (R+1) / S. That is, the training data from line R / S-1 to line (R+1) / S in the target file is used as the training data corresponding to the batch index, and these training data are read for model training.

[0123] Optionally, if the target file matches the file in memory, the target position can be calculated using the method described above. Alternatively, if the target file does not match the file in memory, the target file can be loaded from disk into memory, and the target position can be calculated using the method described above to read the training data for that target position.

[0124] In summary, by using the current batch index, batch size, and the number of batches contained in a single file, the position of the training data corresponding to the batch index in the target file can be calculated, and the training data can be read from the target position. This enables the ordered reading of training data during model training, improving the accuracy and stability of training data reading.

[0125] In one or more embodiments of this application, optionally, the range of values ​​for the batch index can be calculated based on the number of training data sets contained in the plurality of files and the batch size; after model training is performed based on the training data corresponding to the batch index, the batch index to be processed is updated to the current batch index plus one, or, if the current batch index is the maximum value in the range, the updated batch index is set to the minimum value in the range, so as to continue processing the updated batch index.

[0126] Accordingly, if it is determined from the batch index that a training round has been completed based on the current file list, then randomly sorting the file identifiers in the file list may include: if the batch index is the minimum value in the range of values, then it is determined that a training round has been completed based on the current file list, and the file identifiers in the file list are randomly sorted.

[0127] Optionally, the training dataset includes N files, each containing K training data sets. Dividing N*K by the batch size yields the quotient D, which can range from 0 to D-1. The number of training data sets may vary under different training conditions, and the range of the batch index will also change. Understanding the range of the batch index ensures proper training.

[0128] After processing the current batch index, the batch index can be updated. Generally, the batch index can be incremented by one to obtain the updated batch index, which is then processed. The processing procedure for each batch index can be the same, and will not be elaborated here. If the current batch index has reached the maximum value D-1 within the specified range, the next batch index to be processed is the minimum value 0 within the range.

[0129] When the batch index is 0, it indicates that one round of training has been completed based on the current file list, and the file identifiers in the file list can be randomly sorted. Thus, during model training, the batch index increases sequentially from 0. After reaching D-1, the data for one round has been read, and the batch index returns to 0. After re-sorting the file list, the next round of training can begin.

[0130] Figure 5 This is a flowchart illustrating another data processing method provided in an embodiment of this application. Figure 5 As shown, the method may include the following steps.

[0131] Step a: Set the batch index to increment sequentially.

[0132] Optionally, you can use the open-source software TensorFlow to train the model. Setting the shuffle parameter in the model.fit method to False will cause the batch indices to increment sequentially.

[0133] Step b: Scan all files used to train the model and generate a file list.

[0134] Optionally, you can scan all files under the package path and generate a corresponding file list. The file list can be stored in a table.

[0135] Step c: Calculate the length of the full dataset.

[0136] The "full dataset" can refer to the entire training dataset, and its length can be used to determine the range of batch index values. The length can be calculated based on the number of training data sets contained in the multiple files and the batch size. This step allows the calculation of the batch index value range.

[0137] For example, each file contains 128,000 training data entries, the batch size is 128, the number of batches in a single file is 128,000 / 128 = 1,000, and the length of the full dataset corresponding to 780 files is 780 * 1,000 = 780,000. Therefore, the value range of the batch index can be from 0 to 779,999.

[0138] Step d: Determine if the currently input batch index is equal to 0. If it is equal to 0, proceed to step e; otherwise, proceed to step f.

[0139] Optionally, if the batch index is equal to 0, it means that one round of data reading has been completed and the next round of data reading needs to be performed.

[0140] Step e: Randomly sort the file list.

[0141] Before each round of data reading, the file list can be randomly sorted. After random sorting, the file list will change, but the training data contained in each file will remain the same.

[0142] Steps d and e are equivalent to completing: determining the index of the current batch to be processed; if it is determined that a round of training has been completed based on the current file list, then the file identifiers in the file list are randomly sorted.

[0143] Step f: Calculate the file index and cache index based on the batch index.

[0144] Here, the file index can refer to the index corresponding to the target file, while the cache index can refer to the index corresponding to an existing file in memory. Specifically, the file index can be the file identifier corresponding to the file.

[0145] Optionally, the order in which the batch index hits the file in the file list (i.e., the sequence number of the target file in the file list) can be calculated based on the batch index, the number of training data copies contained in a single file, and the batch size.

[0146] Step g: Determine if the file index is equal to the cache index. If yes, proceed to step h; otherwise, proceed to step i.

[0147] Step h: Load the files into memory and sort them randomly.

[0148] Optionally, if the file index is not equal to the cache index, it means that the target file has not yet been cached in memory. In this case, the target file can be loaded into memory. Step i can be executed after step h.

[0149] Steps f to h can achieve the following: determining the target file to which the training data corresponding to the batch index belongs based on the order of file identifiers in the file list; if the target file is inconsistent with the file in memory, then loading the target file from the disk into memory.

[0150] Optionally, loading the target file from the disk into memory and reading the training data corresponding to the batch index from the target file for model training may include: loading the target file from the disk into memory and randomly sorting the training data contained in the target file; and reading the corresponding training data from the randomly sorted target file according to the batch index for model training.

[0151] Optionally, the training data in the target file can be randomly sorted before loading the file into memory, or it can be randomly sorted after loading the file into memory. Alternatively, loading the file and random sorting can be performed simultaneously without any order.

[0152] After random sorting, the order of the training data changes, and consequently, the training data contained in each batch also changes.

[0153] In the aforementioned steps, the file list can be randomly sorted after each round. In this step, the training data within the files is sorted. This way, not only are the files randomly sorted, but the files themselves are also randomly sorted, achieving near-global randomness.

[0154] Furthermore, random sorting between files and random sorting within files are distributed at different times to avoid excessive data reading delays. In particular, it avoids spending too much time on sorting operations after one round of data reading is completed and before reading the next round of data, which can effectively improve the overall efficiency of model training.

[0155] In summary, in this embodiment, not only is the order of each file randomized, but the training data within each file is also randomly sorted when the files are loaded into memory. This achieves near-global randomness, improving the training effect of the model. Furthermore, by setting reasonable timing for random sorting between and within files, the latency of data reading can be reduced, thereby improving the training efficiency of the model.

[0156] Step i: Return the training data corresponding to the batch index.

[0157] Optionally, training data can be read as the data currently needed by the model training software, so that the model training software can train the model based on the read training data.

[0158] Optionally, the training data can be preprocessed before being returned, so that the model training software can update the model parameters based on the preprocessed training data.

[0159] Optionally, preprocessing operations may include: if a feature in the training data is empty, the corresponding value can be filled with the average value or default value of that feature across all training data. For example, if the training data includes product sizes, and a product size in a particular training data point is empty, the product size in that training data point can be set to the average value of the product sizes across all training data.

[0160] By preprocessing the training data before inputting it into the model, the model can process the training data correctly, improving training performance. Furthermore, preprocessing the training data in memory, compared to preprocessing it on disk, reduces the amount of data read from disk, improving data loading efficiency and minimizing resource waste.

[0161] Step j: Update the batch index. Repeat step f until the training termination condition is met.

[0162] Optionally, after training the model based on the training data corresponding to the batch index, the batch index to be processed is updated to the current batch index plus one; or, if the current batch index is the maximum value in the range of values, the updated batch index is set to the minimum value in the range of values, so as to continue processing the updated batch index.

[0163] Optionally, the data reading process and the process of updating model parameters based on the data during model training can be processed in parallel. For example, multiple threads can be set up to implement data reading and parameter updating respectively.

[0164] In summary, the data processing method provided in this embodiment can calculate the value range of the batch index based on the number of training data sets contained in multiple files and the batch size. After training the model based on the training data corresponding to the batch index, the batch index to be processed is updated. Thus, after traversing the batch indexes within the value range in sequence, it can be determined that one round of training has been completed based on the current file list. This method can achieve model training even when the batch index order changes, thereby improving the stability of model training.

[0165] In one or more embodiments of this application, optionally, the method further includes: if it is determined that a round of training has been completed based on the current file list, then resetting the index bias; wherein the index bias is less than the number of training data portions contained in a single file, can be randomly selected from the number of portions, and the index bias is not divisible by the batch size; the index bias is used to indicate the starting position of the first batch of training data in a round.

[0166] Accordingly, determining the sequence number of the file identifier of the target file in the file list based on the number of batches contained in a single file and the batch index may include: determining the sequence number of the file identifier of the target file in the file list based on the number of batches contained in a single file, the batch index, and the index bias.

[0167] Optionally, calculating the position of the training data corresponding to the batch index in the target file based on the batch index, batch size, and the number of batches contained in a single file may include: calculating the position of the training data corresponding to the batch index in the target file based on the batch index, batch size, the number of batches contained in a single file, and the index bias.

[0168] Compared to the aforementioned scheme, when calculating the target file and target position based on the index bias, the only difference is that the starting position of the training data is changed from the first line in the aforementioned scheme to the index bias line, while the rest of the calculation principles and processes remain unchanged.

[0169] For example, the index bias can be set to 100, so the first batch of training data starts from the 100th training data in the first sorted file. Assuming the batch size is 128, the first batch of training data consists of the 100th to the 227th training data. And so on. The training data at the end of the first sorted file may be combined with the training data in the next file to form a batch. In this way, the training data between different files will also be merged.

[0170] In summary, by setting an index bias in each round and calculating the target file and target position based on the index bias, it is possible to achieve random sorting between files, random sorting within files, and fusion of training data from each file. Compared to the scheme in the previous embodiment where the training data between files is not fused, this method can further increase the randomness of the training data, achieve global randomness to a greater extent, and improve the training effect of the model.

[0171] In various embodiments of this application, optionally, when loading a file and reading training data, the entire file can be loaded into memory first and then the required training data can be read. Alternatively, a loading-and-consuming mode can be used, where a new training data is loaded after a training data is used, reducing the time spent waiting for the file to load.

[0172] In one or more embodiments of this application, optionally, the model can be a search recommendation model, which can be used to predict the probability of a user selecting a certain product model; the training data includes user feature information, product model feature information, and labels, wherein the labels are used to indicate whether the user selects the product model.

[0173] Optionally, embodiments of this application can be applied to the home decoration field. In the home decoration field, home decoration design software can be provided to users, allowing them to design the hard and soft furnishings of a house. Furthermore, a search and recommendation model trained using the above method can be used to assess the probability of a user selecting a certain product model, and on the product model display page, product models with higher probabilities can be displayed first, making it easier for users to select product models that interest them.

[0174] Optionally, the product model can be a 2D or 3D product model, specifically a furniture model, home appliance model, or other model used in the home decoration field. The product model information may include at least one of the following: the product model's size, material, style, design, and packaging. User information may include user profile information and / or the user's corresponding context information; the context information may include information currently designed by the user in the home decoration design software, for example, it may include information about existing furniture in the current house.

[0175] In practical applications, a training dataset can be constructed based on users' daily behaviors to train a search and recommendation model. This dataset can then be used to train the model. Within the training dataset, different training data may correspond to the same or different product models, but the entire training dataset should cover all product models in the product model library. After training, the search and recommendation model can be used by home decoration design software to predict in real time the probability of a user selecting each product model.

[0176] Figure 6 This is a schematic diagram illustrating the use of a search recommendation model as provided in an embodiment of this application. Figure 6 As shown, home decoration design software can display a canvas to users, where users can build their own houses and add products to them. As shown in the figure, cabinets and floor lamps have already been designed on the canvas.

[0177] Home decoration design software can also display a search box for users. Users can enter relevant keywords in the search box. For example, if a user wants to add a table to the house, they can enter the keyword "table". There may be many product models that meet the requirements. User profile information, contextual information such as the current house information and the information of the existing furniture in the house, and information of each product model that meets the requirements can be input into the model to obtain the probability of each product model. The product models are then sorted and displayed in descending order of probability. Users can directly select the displayed product model and drag it onto the canvas to add a table to the house.

[0178] In summary, by using the model trained using the aforementioned embodiments, the probability of a user choosing a certain product model can be predicted. This allows for the quick and accurate identification of the product model that the user is most likely to choose, and also places the most likely products at the top of the list to facilitate user selection, thereby improving the efficiency of home decoration design and enhancing the user experience.

[0179] Figure 7 This is a flowchart illustrating a prediction method provided in an embodiment of this application. The method can be applied to the home decoration field, such as... Figure 7 As shown, the method may include:

[0180] Step 701: Based on the search information entered by the user, find at least one corresponding product model from the product model library.

[0181] Optionally, the product model library can store multiple product models. The search information entered by the user can be keywords for the product. Based on the search information, at least one product model that matches the search information can be found from the product model library.

[0182] For example, if a user searches for "table", all table models in the library can be found. If a user searches for "Baroque", all product models in the "Baroque" style can be found.

[0183] Step 702: Based on the user's information and the information of at least one product model, determine the selection probability of each product model based on the search recommendation model, and sort the product models according to the selection probability.

[0184] The search recommendation model is a model trained according to the method described in any of the foregoing embodiments.

[0185] Optionally, the user information and product model information can be input into the search recommendation model to obtain the selection probability of the product model, and then sorted according to the selection probability.

[0186] Step 703: Display at least one product model to the user based on the sorting results.

[0187] Figure 8 This is a schematic diagram illustrating the use of a home decoration scenario model provided in an embodiment of this application. For example... Figure 8 As shown, after a user enters search information, they can select a product model that meets the requirements from the product model library. Based on the search recommendation model, the probability of each product model can be obtained. Then, the product models are sorted according to the probability and displayed to the user.

[0188] In summary, this application embodiment can search for at least one corresponding product model from the product model library based on the search information input by the user. Based on the user's information and the information of the at least one product model, the probability of each product model being selected is determined based on the search recommendation model. The product models are then sorted according to their selection probabilities, and the at least one product model is displayed to the user based on the sorting results. This method can quickly and accurately predict the probability of the user selecting each product model that meets the requirements, and prioritize displaying those with higher probabilities to facilitate user selection, thereby improving the efficiency of home decoration design and enhancing the user experience.

[0189] Corresponding to the above data processing method, this application embodiment also provides a data processing apparatus, the apparatus comprising:

[0190] The first determining module is used to determine the index of the batch to be processed; wherein, the order of the batch indices to be processed changes during model training.

[0191] The sorting module is used to randomly sort the file identifiers in the file list when it is determined that a training round has been completed based on the current file list; wherein the file list includes file identifiers corresponding to multiple files on the disk, and the files contain at least one batch of training data;

[0192] The second determining module is used to determine the target file to which the training data corresponding to the batch index belongs based on the order of the file identifiers in the file list;

[0193] The reading module is used to load the target file from the disk into memory when the target file is inconsistent with the file in memory, and to read the training data corresponding to the batch index from the target file for model training.

[0194] In one or more embodiments of this application, optionally, the second determining module is specifically used for:

[0195] Based on the number of batches contained in a single file and the batch index, determine the sequence number of the file identifier of the target file in the file list;

[0196] The corresponding target file is determined based on the order of the file identifiers in the file list and the sequence number;

[0197] The number of batches contained in a single file is determined by a pre-set batch size and the number of training data sets contained in a single file; the batch size is used to represent the number of training data sets contained in a single batch.

[0198] In one or more embodiments of this application, optionally, the reading module is further configured to:

[0199] If the target file is the same as the file in memory, then the position of the training data corresponding to the batch index in the target file is calculated based on the batch index, batch size, and number of batches contained in a single file;

[0200] The training data corresponding to the batch index is read from the corresponding position of the target file in memory, and the model is trained based on the read training data.

[0201] In one or more embodiments of this application, optionally, the first determining module is further configured to:

[0202] Calculate the range of values ​​for the batch index based on the number of training data sets contained in the multiple files and the batch size;

[0203] After training the model based on the training data corresponding to the batch index, the batch index to be processed is updated to the current batch index plus one. Alternatively, if the current batch index is the maximum value in the range of values, the updated batch index is set to the minimum value in the range of values, so as to continue processing the updated batch index.

[0204] Accordingly, the sorting module is specifically used for:

[0205] If the batch index is the minimum value in the range, it is determined that one round of training has been completed based on the current file list, and the file identifiers in the file list are randomly sorted.

[0206] In one or more embodiments of this application, optionally, when the reading module loads the target file from the disk into memory and reads the training data corresponding to the batch index from the target file for model training, it is specifically used for:

[0207] The target file in the disk is loaded into memory, and the training data contained in the target file is randomly sorted.

[0208] Based on the batch index, the corresponding training data is read from the randomly sorted target file for model training.

[0209] In one or more embodiments of this application, optionally, the model is used to predict the probability that a user selects a certain product model; the training data includes user feature information, product model feature information, and labels, the labels being used to indicate whether the user selects the product model; the reading module is further used to:

[0210] Before training the model, the training dataset used to train the model is downloaded to the disk and the training dataset is divided into the multiple files for storage;

[0211] The training dataset is presented to personnel involved in model training so that they can process abnormal data in the training dataset; and / or, abnormal training data in the training dataset is detected and processed based on the median of each feature information in the training dataset.

[0212] Corresponding to the above prediction method, this application embodiment also provides a prediction device, including:

[0213] The search module is used to find at least one corresponding product model from the product model library based on the search information entered by the user.

[0214] The third determining module is used to determine the selection probability of each product model based on the user's information and the information of the at least one product model, and to sort the product models according to the selection probability.

[0215] The display module is used to display at least one product model to the user based on the sorting results;

[0216] The search recommendation model is a model trained according to any one of the methods described in the first aspect.

[0217] The data processing apparatus and prediction apparatus provided in this application embodiment can be used to perform the above-described... Figures 1 to 8 The technical solutions of the embodiments shown are similar in principle and in effect, and will not be described again here.

[0218] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 9 As shown, the electronic device in this embodiment may include:

[0219] At least one processor 901; and

[0220] Memory 902 communicatively connected to the at least one processor;

[0221] The memory 902 stores instructions that can be executed by the at least one processor 901, which, when executed by the at least one processor 901, cause the electronic device to perform the method described in any of the above embodiments.

[0222] Alternatively, the memory 902 can be either standalone or integrated with the processor 901.

[0223] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the foregoing embodiments, and will not be repeated here.

[0224] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method described in any of the foregoing embodiments.

[0225] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods described in any of the foregoing embodiments.

[0226] The collection, storage, use, processing, transmission, provision, and disclosure of user data and other information involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0227] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0228] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application.

[0229] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor. The memory may include high-speed RAM, and may also include non-volatile memory (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk, or optical disc, etc.

[0230] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0231] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.

[0232] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0233] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0234] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0235] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: Determine the current batch index to be processed; the order of the batch indices to be processed changes during model training. If it is determined from the batch index that a training round has been completed based on the current file list, then the file identifiers in the file list are randomly sorted; wherein, the file list includes file identifiers corresponding to multiple files on the disk, and the files contain training data for at least one batch; Based on the order of file identifiers in the file list, determine the target file to which the training data corresponding to the batch index belongs; If the target file is inconsistent with the file in memory, the target file in the disk is loaded into memory, and the training data contained in the target file is randomly sorted. Based on the batch index, the training data corresponding to the batch index is read from the randomly sorted target file for model training; The method further includes: Calculate the range of values ​​for the batch index based on the number of training data sets contained in the multiple files and the batch size; After training the model based on the training data corresponding to the batch index, the batch index to be processed is updated to the current batch index plus one. Alternatively, if the current batch index is the maximum value in the range of values, the updated batch index is set to the minimum value in the range of values, so as to continue processing the updated batch index. Accordingly, if it is determined from the batch index that a training round has been completed based on the current file list, it includes: If the batch index is the minimum value in the range, then it is determined that one round of training has been completed based on the current file list.

2. The method according to claim 1, characterized in that, Based on the order of file identifiers in the file list, determine the target file to which the training data corresponding to the batch index belongs, including: Based on the number of batches contained in a single file and the batch index, determine the sequence number of the file identifier of the target file in the file list; The corresponding target file is determined based on the order of the file identifiers in the file list and the sequence number; The number of batches contained in a single file is determined by a pre-set batch size and the number of training data sets contained in a single file; the batch size is used to represent the number of training data sets contained in a single batch.

3. The method according to claim 2, characterized in that, Also includes: If the target file is the same as the file in memory, then the position of the training data corresponding to the batch index in the target file is calculated based on the batch index, batch size, and number of batches contained in a single file; The training data corresponding to the batch index is read from the corresponding position of the target file in memory, and the model is trained based on the read training data.

4. The method according to any one of claims 1-3, characterized in that, The model is used to predict the probability that a user will select a certain product model; the training data includes user feature information, product model feature information, and labels, the labels being used to indicate whether the user has selected the product model. Before training the model, the following is also included: The training dataset used to train the model is downloaded to the disk, and the training dataset is divided into the multiple files for storage; The training dataset is presented to personnel involved in model training so that they can process abnormal data in the training dataset; and / or, abnormal training data in the training dataset is detected and processed based on the median of each feature information in the training dataset.

5. A prediction method, characterized in that, include: Based on the search information entered by the user, at least one corresponding product model is found from the product model library; Based on the user's information and the information of at least one product model, the probability of each product model being selected is determined based on the search recommendation model, and the product models are ranked according to the probability of selection. Display at least one product model to the user based on the sorting results; The search recommendation model is a model trained according to any one of claims 1-4.

6. An electronic device, characterized in that, include: At least one processor; as well as A memory that is communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, cause the electronic device to perform the method according to any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-5.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-5.

Images