A data processing method, device, equipment and readable storage medium

Abstract

The present disclosure provides a data processing method, device and equipment and readable storage medium, wherein the data processing method comprises: classifying each to-be-processed data of a to-be-processed data set based on a preset classification condition to obtain a first data set composed of first data of the same classification; deduplicating dependent data of each first data to obtain a second data set composed of second data after deduplication; performing information aggregation on each first data to obtain a third data set composed of third data after information aggregation, wherein the third data maps loading information and preview information of the first data; and compressing the first data set, the second data set and the third data set to obtain an aggregated file comprising a plurality of separately loadable blocks, wherein each block corresponds to each first data. The present disclosure can improve data uploading efficiency and loading speed, and reduce loading resource consumption.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically to a data processing method, apparatus, device, and readable storage medium. Background Technology

[0002] In UGC content creation scenarios, users can upload their created material files to the platform's cloud server for display, and other users can download the resources from the platform.

[0003] However, user-uploaded media files typically include multiple data points. Users need to upload each individual data point to the platform one by one, which results in low upload efficiency and high computational resource consumption for large media files. Summary of the Invention

[0004] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a data processing method, apparatus, device, and medium that can improve data upload efficiency and loading speed, and reduce loading resource consumption.

[0005] According to one aspect of this disclosure, a data processing method is provided, the method comprising: classifying each piece of data to be processed in a dataset based on preset classification conditions to obtain a first dataset composed of first data of the same classification; deduplicating dependent data of each piece of first data to obtain a second dataset composed of deduplicated second data; aggregating information of each piece of first data to obtain a third dataset composed of aggregated third data, wherein the third data maps loading information and preview information of the first data; and compressing the first dataset, the second dataset, and the third dataset to obtain an aggregated file comprising multiple individually loadable blocks, wherein each block corresponds to each piece of first data.

[0006] According to another aspect of this disclosure, a data processing apparatus is provided, the apparatus comprising: a first data set acquisition module, configured to classify each piece of data to be processed in a data set to be processed based on preset classification conditions, to obtain a first data set composed of first data of the same classification; a second data set acquisition module, configured to deduplicate dependent data of each piece of first data, to obtain a second data set composed of deduplicated second data; a third data set acquisition module, configured to aggregate information of each piece of first data, to obtain a third data set composed of aggregated third data, wherein the third data maps loading information and preview information of the first data; and an aggregate file acquisition module, configured to compress the first data set, the second data set, and the third data set to obtain an aggregate file comprising multiple individually loadable blocks, wherein each block corresponds to each piece of first data.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to perform the data processing method described above.

[0008] According to another aspect of this disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the above-described data processing method.

[0009] The data processing method and apparatus provided in this embodiment can classify each piece of data to be processed in a dataset based on preset classification conditions to obtain a first data set composed of first data of the same classification; deduplicate the dependent data of each piece of first data to obtain a second data set composed of deduplicated second data; aggregate information of each piece of first data to obtain a third data set composed of aggregated third data, wherein the third data maps the loading information and preview information of the first data; and compress the first data set, the second data set, and the third data set to obtain an aggregated file including multiple individually loadable blocks, wherein each block corresponds to each piece of first data.

[0010] By employing the above technical solution, each piece of data to be processed in the dataset can be classified based on preset classification conditions, resulting in a first data set composed of first data of the same classification. Then, the dependent data of each first data can be deduplicated, resulting in a second data set composed of deduplicated second data. Next, information aggregation can be performed on each first data, resulting in a third data set composed of aggregated third data. Finally, the first, second, and third data sets can be compressed to obtain an aggregate file containing multiple individually loadable blocks. Because the aggregate file of this disclosure contains deduplicated dependent data of multiple first data of the same type, as well as aggregate information mapped with loading and preview information, it enables the same type of data to be processed and its related data to be quickly and completely compressed and uploaded together. Furthermore, each individually loadable block in the aggregate information corresponds to one piece of first data, allowing each piece of first data to be loaded individually through the loading and preview information mapped in the aggregate information. Therefore, the above technical solution can significantly improve the efficiency and speed of data upload and reduce the computational resources required for data loading.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram illustrating an application scenario of a data classification system provided in an embodiment of this disclosure;

[0014] Figure 2 This is a schematic flowchart of a data processing method provided in an embodiment of this disclosure;

[0015] Figure 3 This is a schematic diagram illustrating an embodiment of obtaining a first data set provided in this disclosure;

[0016] Figure 4 This is a schematic diagram illustrating a display of a list of preview images provided in an embodiment of this disclosure;

[0017] Figure 5 This is a schematic diagram illustrating the generation of third data according to an embodiment of this disclosure;

[0018] Figure 6This is a schematic diagram illustrating a display of aggregated files provided in an embodiment of this disclosure;

[0019] Figure 7 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this disclosure;

[0020] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0021] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] This disclosure provides a data processing method, apparatus, device, and readable storage medium. The device can be an electronic device, and the readable storage medium can be a computer-readable storage medium. The data processing apparatus can be integrated into an electronic device, which can be a server, a terminal, or other similar device.

[0023] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms.

[0024] The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and the server can be connected directly or indirectly through wired or wireless communication, which is not limited herein.

[0025] For example, such as Figure 1As shown, server 11 can classify each piece of data to be processed in the dataset based on preset classification conditions, resulting in a first data set composed of first data of the same classification; deduplicate the dependent data of each first data to obtain a second data set composed of deduplicated second data; aggregate information for each first data to obtain a third data set composed of aggregated third data, wherein the third data maps the loading information and preview information of the first data; compress the first data set, the second data set, and the third data set to obtain an aggregate file including multiple individually loadable blocks, wherein each block corresponds to each first data. The data to be processed sent by terminal 10 can be compressed into an aggregate file by performing the above steps.

[0026] It should be noted that the steps performed by server 11 can also be performed through terminal 10 to upload the aggregated file to server 11.

[0027] In this disclosure, "multiple" refers to two or more. The terms "first" and "second," etc., used in this disclosure are for distinguishing descriptions and should not be construed as implying relative importance.

[0028] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0029] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of the present disclosure. The method can be executed by a data processing device, which can be implemented using software and / or hardware, and is generally integrated into an electronic device. Figure 2 As shown, the method mainly includes:

[0030] Step 101: Based on the preset classification conditions, classify each piece of data to be processed in the dataset to be processed, and obtain a first dataset consisting of first data with the same classification.

[0031] In this context, the data to be processed can be raw, unprocessed data. Taking a UGC (User Generated Content) display scenario as an example, users can upload completed material files to the cloud, websites, apps, and other channels for display. These material files can include 3D models, videos, images, audio, and other data. Before uploading, users can categorize, compress, and modify the material files. The raw material files that have not undergone these processes constitute the data to be processed in this scenario.

[0032] In some embodiments, different data to be processed can be classified to obtain a first data set consisting of first data of the same classification. It is understood that the material file to be uploaded by the user may include many different types of sub-files. For example, if the material file to be uploaded by the user is a game model, then the material file may include multiple game model data, image data, and mesh data, etc.

[0033] It's understandable that in scenarios involving uploading data to be processed, large-volume media files often contain numerous sub-files, requiring users to process each file individually, resulting in a large total number of files and a significant amount of memory usage. For example, if a user uploads 100 files, they would need to perform 100 operations and wait for each of the 100 files to be uploaded sequentially. Therefore, different types of data to be processed can be categorized to facilitate subsequent processing of the first data within the same category, thereby improving data processing efficiency.

[0034] Optionally, step 101 may include:

[0035] Obtain the file extension of each piece of data to be processed;

[0036] Multiple data items with the same file extension are identified as the first data, thus obtaining the first data set.

[0037] The preset classification criteria can be to select data with the same file extension as the first data to be classified. It's understandable that different types of data are generally stored with different file extensions, such as model data with the extension ".prefab", image data with the extension ".png", and grid data with the extension ".msh".

[0038] Figure 3 This is a schematic diagram illustrating an embodiment of obtaining a first data set. For example... Figure 3 As shown, after obtaining multiple first data entries based on preset classification criteria with the same file extension, these first data entries can be moved to their respective folders. For example, after filtering out multiple first data entries with the extension ".prefab", these first data entries can be moved to a folder named "Prefabs" to obtain a set of first data entries. Subsequent first data entries with the extension ".prefab" can also be moved to this folder. Correspondingly, for first data entries with different file extensions, different folders can be created based on their respective file extensions to store different types of first data entries.

[0039] It should be noted that this embodiment does not impose restrictions on the naming rules, format, and specific name of the folder containing the first data set.

[0040] Step 102: Deduplicate the dependent data of each of the first data to obtain a second data set composed of the deduplicated second data.

[0041] In this context, dependent data refers to sub-data of the first data. For example, if the first data is model data, then its dependent data could be its corresponding image data and mesh data, such as the rendering images and meshes used to render the model. It can be understood that dependent data is equivalent to sub-files of the source file. For independent data to be processed that does not depend on other data, there is no corresponding dependent data, such as image data, video data, and mesh data.

[0042] Optionally, step 102 may include:

[0043] Obtain the dependency identifier for each of the first data items;

[0044] Based on each of the dependency identifiers, the dependency data for each of the first data items is obtained;

[0045] Deduplication is performed on multiple identical dependency data to obtain unique dependency data that is retained after deduplication.

[0046] The unique dependent data and the dependent data that do not have the same data are jointly identified as the second data, and multiple sets of the second data are organized to obtain the second data set.

[0047] In some embodiments, dependency analysis can be performed on each piece of first data to obtain a dependency identifier for each piece of first data. The dependency identifier can be a file ID of the dependent data, which can be represented by numbers, letters, etc., for example, a file ID could be "350722". In some embodiments, the dependent data of the first data can be obtained based on the file ID. For example, the first data in JSON format can be read into memory, then the file information in memory can be read, and the dependency identifier recorded in the file information can be used to obtain all image data and mesh data corresponding to the model data of the first data.

[0048] In some embodiments, after obtaining the dependency data for each first data set, multiple identical dependency data sets can be deduplicated to obtain a unique dependency data set. For example, for two model class first data sets with the .prefab extension, if they share the same rendered texture map, resulting in identical dependency data (e.g., a dependency data set named "1.png"), then these two identical dependency data sets can be deduplicated to obtain a unique dependency data set named "1.png". It should be noted that this deduplication process does not limit the number of identical dependency data sets; only one unique dependency data set is retained after deduplication.

[0049] It is understandable that after obtaining the dependent data of each first data, there are dependent data with the same data and dependent data with different data. If multiple first data reference the same dependent data, such as the same image data B, then when the above multiple first data exist in the same scenario, it will cause image data B to be loaded multiple times in this scenario. Therefore, by deduplicating the same dependent data, no matter how many first data with the same dependent data are loaded at the same time, the same dependent data will only be loaded once in the end, thereby improving the data loading efficiency and reducing the computing resource consumption required for loading data.

[0050] In some embodiments, all dependent data can be deduplicated by default, and dependent data that does not have the same data can be directly identified as the second data. In some embodiments, a data identification operation can be used to deduplicate only dependent data that has the same data to obtain unique dependent data, skipping the deduplication of dependent data that does not have the same data, and identifying both types of data as the second data.

[0051] In some embodiments, a dependency data folder (e.g., a Dependencies folder) at the same level as the first data folder (e.g., a Prefabs folder) can be created, and the second data can be stored in this dependency data folder to obtain the second data set. It should be noted that if the first data does not have any dependency data, there is no need to create a corresponding dependency data folder.

[0052] Step 103: Aggregate information for each of the first data to obtain a third data set composed of the aggregated third data.

[0053] The third data can map the loading and preview information of the first data. Loading information refers to the data used to load the data, such as the data loading file, loading path, and loading rules. Preview information refers to the data used to preview the data, such as the preview file, preview path, and preview rules.

[0054] It is understandable that, in order to achieve successful loading and previewing of the first data, the loading and preview information of the first data can be aggregated to obtain third data that maps the loading and preview information of the first data. This allows subsequent processing of the third data, such as compression, uploading, or merging with other first data, to ensure that the resulting files will carry the loading and preview information of the first data. As a result, the processed files can accurately load and preview the files containing the first data.

[0055] Optionally, step 103 may include:

[0056] Remove the suffix from the name of the first data to obtain the first name;

[0057] Configure the loading information of the first data, and map the loading information to the first name to obtain the first mapping result;

[0058] Configure preview information for the first data, and map the preview information to the first name to obtain the second mapping result;

[0059] Write the first mapping result and the second mapping result into the target file to obtain the third data;

[0060] The above steps are repeated for each of the first data points, and each of the third data points is written into the target file to obtain the third data set.

[0061] In some embodiments, the name of the first data can be desuffed to obtain the first name. For example, desuffing the name of the first data A.prefab removes the suffix ".prefab" to obtain the first name "A". Accordingly, for Figure 3 The first data in the file, “C_CQ_shibing_mao.pr efab”, is processed by removing the suffix to obtain the first name “C_CQ_shibing_mao”.

[0062] In some embodiments, loading information for the first data can be configured. Specifically, the storage location of the first data can be obtained, the path of the storage location can be defined as the loading path of the first data, and then the loading path can be concatenated with the name of the first data to obtain the loading information. Taking the first data A.prefab as an example, assuming that the storage location of the first data is the Prefabs folder in the assets folder, the loading path of the first data can be obtained as assets / Prefabs, and then the loading path can be concatenated with the name of the first data to obtain the loading information "assets / Prefabs / A.prefab".

[0063] Optionally, the step "configure the loading information of the first data and map the loading information to the first name to obtain a first mapping result" includes:

[0064] Obtain the loading path of the first data and the name of the first data;

[0065] By concatenating the loading path and the name of the first data, the loading information is obtained;

[0066] Establish a mapping relationship between the loaded information and the first name to obtain the first mapping result.

[0067] In some embodiments, loading information can be mapped to a first name to obtain a first mapping result. Specifically, the first name can be concatenated with the loading information to obtain the first mapping result. For example, concatenating the first name A with the aforementioned loading information assets / Prefabs / A.prefab yields the first mapping result "A: assets / Prefabs / A.prefab", indicating that the first mapping result provides loading information for the first data A.prefab. That is, when the first data A.prefab needs to be loaded, the file A.prefab of the first data can be loaded from the assets / Prefabs folder.

[0068] Optionally, the step "configure preview information for the first data and map the preview information to the first name to obtain the second mapping result" includes:

[0069] Obtain the preview path of the first data and the preview image of the first data;

[0070] By concatenating the preview path and the name of the preview image, the preview information is obtained;

[0071] Establish a mapping relationship between the preview information and the first name to obtain a second mapping result.

[0072] In some embodiments, a preview image corresponding to the first data can be obtained first. Continuing with the first data A.prefab as an example, the A.prefab file can be loaded into memory, the rendering result of the file displayed in memory as byte content can be obtained, the obtained byte content can be written to an empty file and named with an image format extension, such as "A.png", to obtain the preview image of the first data.

[0073] Figure 4 This is a schematic diagram illustrating a display of a preview image list provided in an embodiment of this disclosure. For example... Figure 4 As shown, there are three preview image files in the preview image list. Each preview image file consists of a thumbnail and a filename. For example, the file C_CQ_shibing_mao.png can be a preview image of a soldier wearing a spear in the game, and the corresponding thumbnail can be a small image of a soldier model with that feature.

[0074] In some embodiments, the method for obtaining the preview path and the second mapping result is similar to the method for obtaining the loading path and the first mapping result. Specifically, the storage location of the preview image of the first data can be obtained, the path of the storage location can be defined as the preview path of the first data, and then the preview path can be concatenated with the name of the first data to obtain the preview information. Taking the first data A.prefab as an example, assuming that the storage location of the preview image of the first data is the Prefabs folder in the Preview folder, the preview path of the first data can be obtained as Preview, and then the loading path can be concatenated with the name of the preview image of the first data to obtain the loading information "Preview / A.png".

[0075] In some embodiments, a mapping relationship between preview information and a first name can be established to obtain a second mapping result. Specifically, the first name and preview information can be concatenated to obtain the second mapping result. For example, concatenating the first name A with the above preview information Preview / A.png yields the second mapping result "A:Preview / A.png", indicating that the second mapping result provides preview information for the first data A.prefab. That is, when the first data A.prefab needs to be previewed, the preview image file A.png of the first data can be previewed from the Preview folder.

[0076] In application, for example after the first data is uploaded to the server, users can hover their mouse over any one or more files in the preview image list to bring up a preview image of that first data. Users can also drag and drop preview image files into the scene of the display interface to display the corresponding model of the first data in that interface.

[0077] In some embodiments, the first mapping result and the second mapping result can be written into a target file to obtain the third data, and the above steps can be repeated for each piece of first data to write each piece of third data into a target file to obtain a third data set.

[0078] In some embodiments, the first mapping result and the second mapping result can be written into a target file in JSON format to obtain third data in JSON format. Figure 5 This is a schematic diagram illustrating the generation of third-party data provided in an embodiment of this disclosure, such as... Figure 5 As shown, the first data is... Figure 3Taking the three model files C_CQ_shibing_mao.prefab, C_CQ_shibing_chudi.prefab, and C_CQ_shibing_ge.prefab as examples, the loading information of these three first data sets is written into the prefaaddressMap field, and the preview information of these three first data sets is written into the prefaabPreviewMap field, so that this third data set can map the loading information and preview information of the three first data sets.

[0079] for Figure 5 Other fields in the dataset include: the `name` field, which represents the loading prefix for the third-party data set, essentially acting as its unique identifier on the server; and the `name` field's content, such as "251051664285428", which is appended to the `assets` field to form the full path for loading the third-party data set. The `version` field represents the version of the third-party data set, and `unityVersion` represents the corresponding 3D engine version. In practical application logic, the `version` and `unityVersion` fields are used to collect information about the third-party data set and distinguish its version. The `platform` field represents the operating system platforms that the third-party data set is compatible with. For example, data from macOS, Windows, iOS, and Android are not interchangeable; the final user will receive different asset packages based on their operating system platform.

[0080] Step 104: Compress the first data set, the second data set, and the third data set to obtain an aggregate file containing multiple individually loadable blocks.

[0081] In some embodiments, the first data set, the second data set, and the third data set can be compressed using the AssetBundle compression algorithm to obtain an aggregate file that includes multiple individually loadable blocks.

[0082] Each block can correspond to each first data. Figure 6 This is a schematic diagram illustrating a display of aggregated files provided in an embodiment of this disclosure, such as... Figure 6 As shown, after obtaining the aggregate file with the .bundle extension, the aggregate file can be stored as a virtual model. The aggregate file can be divided into different blocks according to the number of first data. That is, each block corresponds to a first data and stores the information of the second and third data corresponding to the first data.

[0083] Optionally, the method disclosed herein may also include:

[0084] Receive a load request for the first data;

[0085] According to the loading request, read the block where the first data is located in the aggregate file;

[0086] The block corresponding to the first data is added to the loading memory to load the first data.

[0087] It's understandable that after obtaining the aggregated file using the AssetBundle compression algorithm, the server can leverage this algorithm to load different blocks on demand, loading specific first data based on the user's loading requirements. For example, if a user requests to load the first data C_CQ_shibing_chudi.prefab, the server can obtain the loading request and read the block of that first data accordingly. For instance, if the first data block is the 10th block, then the 10th block from the aggregated file can be added to the loading memory to load the first data. Other blocks of the first data will not be read or added to the loading memory, thereby improving data loading speed and efficiency while reducing the computational resources required for data loading.

[0088] Therefore, this disclosure can classify each piece of data to be processed in the dataset based on preset classification conditions, obtaining a first data set composed of first data of the same classification. Then, the dependent data of each first data can be deduplicated, resulting in a second data set composed of deduplicated second data. Next, information aggregation can be performed on each first data, resulting in a third data set composed of aggregated third data. Finally, the first, second, and third data sets can be compressed to obtain an aggregate file containing multiple individually loadable blocks. Because the aggregate file of this disclosure contains deduplicated dependent data of multiple first data of the same type, as well as aggregate information mapped with loading and preview information, it enables the same type of data to be processed and its related data to be quickly and completely compressed and uploaded together. Furthermore, each individually loadable block in the aggregate information corresponds to one piece of first data, allowing each piece of first data to be loaded individually through the loading and preview information mapped in the aggregate information. Thus, the above technical solution can significantly improve the efficiency and loading speed of data upload and reduce the computing resources required for data loading.

[0089] The method described in the above embodiments will be further described in detail below.

[0090] Corresponding to the aforementioned data processing method, this disclosure also provides a data processing apparatus. Figure 7This is a schematic diagram of a data processing apparatus provided in an embodiment of the present disclosure. The apparatus can be implemented by software and / or hardware and is generally integrated into an electronic device. Figure 7 As shown, the data processing device 300 includes:

[0091] The first data set acquisition module 301 is used to classify each piece of data to be processed in the data set to be processed based on preset classification conditions, so as to obtain a first data set composed of first data with the same classification.

[0092] The second data set acquisition module 302 is used to deduplicate the dependent data of each of the first data to obtain a second data set composed of the deduplicated second data.

[0093] The third data set acquisition module 303 is used to aggregate information on each of the first data to obtain a third data set composed of the aggregated third data, wherein the third data maps the loading information and preview information of the first data;

[0094] The aggregate file acquisition module 304 is used to compress the first data set, the second data set and the third data set to obtain an aggregate file including multiple individually loadable blocks, wherein each block corresponds to each first data set.

[0095] In some embodiments, the third data set acquisition module 303 includes:

[0096] The suffix removal submodule is used to remove the suffixes from the name of the first data to obtain the first name;

[0097] The first mapping submodule is used to configure the loading information of the first data and map the loading information to the first name to obtain the first mapping result;

[0098] The second mapping submodule is used to configure the preview information of the first data and map the preview information to the first name to obtain the second mapping result;

[0099] The data writing submodule is used to write the first mapping result and the second mapping result into the target file to obtain the third data;

[0100] The third data set acquisition submodule is used to repeatedly execute the above steps for each first data, write each third data into the target file, and obtain the third data set.

[0101] In some implementations, the first mapping submodule is used for:

[0102] Obtain the loading path of the first data and the name of the first data;

[0103] By concatenating the loading path and the name of the first data, the loading information is obtained;

[0104] Establish a mapping relationship between the loaded information and the first name to obtain the first mapping result.

[0105] In some implementations, the second mapping submodule is used for:

[0106] Obtain the preview path of the first data and the preview image of the first data;

[0107] By concatenating the preview path and the name of the preview image, the preview information is obtained;

[0108] Establish a mapping relationship between the preview information and the first name to obtain a second mapping result.

[0109] In some embodiments, the data processing apparatus of this disclosure further includes:

[0110] A load request receiving module is used to receive a load request for the first data;

[0111] The block reading module is used to read the block in the aggregate file where the first data is located, according to the loading request;

[0112] The first data loading module is used to add the block corresponding to the first data to the loading memory in order to load the first data.

[0113] In some implementations, the second data set acquisition module 302 is used for:

[0114] Obtain the dependency identifier for each of the first data items;

[0115] Based on each of the dependency identifiers, the dependency data for each of the first data items is obtained;

[0116] Deduplication is performed on multiple identical dependency data to obtain unique dependency data that is retained after deduplication.

[0117] The unique dependent data and the dependent data that do not have the same data are jointly identified as the second data, and multiple sets of the second data are organized to obtain the second data set.

[0118] In some implementations, the first data set acquisition module 301 is used for:

[0119] Obtain the file extension of each piece of data to be processed;

[0120] Multiple data items with the same file extension are identified as the first data, thus obtaining the first data set.

[0121] Therefore, this disclosure can classify each piece of data to be processed in the dataset based on preset classification conditions, obtaining a first data set composed of first data of the same classification. Then, the dependent data of each first data can be deduplicated, resulting in a second data set composed of deduplicated second data. Next, information aggregation can be performed on each first data, resulting in a third data set composed of aggregated third data. Finally, the first, second, and third data sets can be compressed to obtain an aggregate file containing multiple individually loadable blocks. Because the aggregate file of this disclosure contains deduplicated dependent data of multiple first data of the same type, as well as aggregate information mapped with loading and preview information, it enables the same type of data to be processed and its related data to be quickly and completely compressed and uploaded together. Furthermore, each individually loadable block in the aggregate information corresponds to one piece of first data, allowing each piece of first data to be loaded individually through the loading and preview information mapped in the aggregate information. Thus, the above technical solution can significantly improve the efficiency and loading speed of data upload and reduce the computing resources required for data loading.

[0122] The data processing apparatus provided in this disclosure can execute the data processing method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method.

[0123] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device embodiments can be referred to the corresponding process in the method embodiments, and will not be repeated here.

[0124] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0125] Exemplary embodiments of this disclosure also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the electronic device to perform a method according to an embodiment of this disclosure.

[0126] Exemplary embodiments of this disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to embodiments of this disclosure.

[0127] Exemplary embodiments of this disclosure also provide a computer program product, including a computer program, wherein, when executed by a processor of a computer, the computer program is used to cause the computer to perform a method according to an embodiment of this disclosure.

[0128] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0129] Furthermore, embodiments of this disclosure can also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the data processing method provided in embodiments of this disclosure. The computer-readable storage medium can be any combination of one or more readable media. A readable medium can be a readable signal medium or a readable storage medium. A readable storage medium can be, for example, including but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0130] refer to Figure 8 The present invention describes a structural block diagram of an electronic device 300 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0131] like Figure 8As shown, the electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. The RAM 303 may also store various programs and data required for the operation of the device 300. The computing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0132] Multiple components in electronic device 300 are connected to I / O interface 305, including: input unit 306, output unit 307, storage unit 308, and communication unit 309. Input unit 306 can be any type of device capable of inputting information to electronic device 300. Input unit 306 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 307 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 308 may include, but is not limited to, disk and optical disk. Communication unit 309 allows electronic device 300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0133] The computing unit 301 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 301 performs the various methods and processes described above. For example, in some embodiments, the data processing methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 300 via ROM 302 and / or communication unit 309. In some embodiments, the computing unit 301 can be configured to perform data processing methods by any other suitable means (e.g., by means of firmware).

[0134] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0135] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0136] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0137] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0138] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0139] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0140] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0141] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method, characterized in that, The method includes: Based on the file extension of the data to be processed, each data to be processed in the data set is classified to obtain a first data set consisting of first data of the same classification. Each piece of dependent data of the first data is deduplicated to obtain a second data set consisting of the deduplicated second data. The second data set includes: unique dependent data obtained by deduplicating multiple identical dependent data, and dependent data that does not have any duplicate data. For each piece of the first data: the name of the first data is desufficient to obtain a first name; the loading information of the first data is configured and mapped to the first name to obtain a first mapping result; the preview information of the first data is configured and mapped to the first name to obtain a second mapping result; the first mapping result and the second mapping result are written to a target file to obtain third data; the third data of each piece of the first data in the first data set constitutes a third data set, wherein the third data maps the loading information and preview information of the first data; The first data set, the second data set, and the third data set are compressed to obtain an aggregate file containing multiple individually loadable blocks, wherein each block corresponds to each of the first data sets.

2. The data processing method according to claim 1, characterized in that, Configure the loading information of the first data and map the loading information to the first name to obtain a first mapping result, including: Obtain the loading path of the first data and the name of the first data; By concatenating the loading path and the name of the first data, the loading information is obtained; Establish a mapping relationship between the loaded information and the first name to obtain the first mapping result.

3. The data processing method according to claim 1, characterized in that, Configure preview information for the first data and map the preview information to the first name to obtain a second mapping result, including: Obtain the preview path of the first data and the preview image of the first data; By concatenating the preview path and the name of the preview image, the preview information is obtained; Establish a mapping relationship between the preview information and the first name to obtain a second mapping result.

4. The data processing method according to claim 1, characterized in that, After obtaining an aggregate file comprising multiple individually loadable blocks, the method includes: Receive a load request for the first data; According to the loading request, read the block where the first data is located in the aggregate file; The block corresponding to the first data is added to the loading memory to load the first data.

5. The data processing method according to claim 1, characterized in that, The step of deduplicating the dependent data of each of the first data to obtain a second data set composed of the deduplicated second data includes: Obtain the dependency identifier for each of the first data items; Based on each of the dependency identifiers, the dependency data for each of the first data items is obtained; Deduplication is performed on multiple identical dependency data to obtain unique dependency data that is retained after deduplication. The unique dependent data and the dependent data that do not have the same data are jointly identified as the second data, and multiple sets of the second data are organized to obtain the second data set.

6. The data processing method according to claim 1, characterized in that, The classification of each piece of data to be processed in the dataset based on its file extension includes: Obtain the file extension of each piece of data to be processed; Multiple data items with the same file extension are identified as the first data, thus obtaining the first data set.

7. A data processing apparatus, characterized in that, include: The first data set acquisition module is used to classify each piece of data to be processed in the data set based on the suffix of the data to be processed, so as to obtain a first data set composed of first data with the same classification. The second data set acquisition module is used to deduplicate the dependent data of each of the first data to obtain a second data set composed of the deduplicated second data. The second data set includes: unique dependent data obtained by deduplicating multiple identical dependent data, and dependent data that do not have the same data. The third data set acquisition module is used for each piece of the first data to: remove the suffix from the name of the first data to obtain a first name; configure the loading information of the first data and map the loading information to the first name to obtain a first mapping result; configure the preview information of the first data and map the preview information to the first name to obtain a second mapping result; write the first mapping result and the second mapping result to a target file to obtain third data; the third data of each piece of the first data in the first data set constitutes the third data set, wherein the third data maps the loading information and preview information of the first data; An aggregated file acquisition module is used to compress the first data set, the second data set, and the third data set to obtain an aggregated file comprising multiple individually loadable blocks, wherein each block corresponds to each of the first data sets.

8. An electronic device, characterized in that, The electronic device includes: Processor; and Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform the data processing method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the data processing method according to any one of claims 1 to 6.

Images