Data processing method and device, equipment and storage medium

Abstract

The invention provides a data processing method and device, equipment and a storage medium. The method comprises the steps that a data set is divided into at least two sub-data sets; the data set comprises a plurality of data blocks, and each data block comprises a storage initial address and multiple rows of data; analyzing the at least two sub-data sets in parallel; wherein the parallel analysis of the at least two sub-data sets comprises the following steps of: dividing a target data block in the plurality of data blocks into a first sub-data set and a second sub-data set in response to a division position of the data set, dividing the target data block into a first part and a second part, and analyzing the first sub-data set and the second sub-data set in parallel; the first part comprises a storage initial address of the target data block; and in response to the completion of analysis of the first sub-data set and the second sub-data set, determining a storage initial address corresponding to the second part according to the storage initial address and the termination address corresponding to the first part obtained by analysis. According to the method and the device, the data processing speed and efficiency are improved by analyzing the plurality of data blocks in parallel.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to a data processing method, device, equipment and storage medium. Background Art

[0002] In high-performance chips, memory is a key part of performance, responsible for storing large amounts of data, instructions, cache information, etc., while ensuring that the chip can perform various tasks efficiently and accurately. With the continuous development of technology, the scale of memory is also expanding. This ultra-large-scale memory places extremely high demands on the performance of the simulator, and the simulator needs to fully support this large-scale memory structure.

[0003] However, for large memory numerical files, the traditional serial parsing method is intuitive but time-consuming and cannot fully utilize the performance advantages of multi-core CPUs, resulting in low efficiency in the simulator's data parsing process. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a data processing method, device, equipment and storage medium.

[0005] As one aspect of the present application, a data processing method is provided, comprising:

[0006] Dividing a data set into at least two sub-data sets; the data set includes a plurality of data blocks, each of the data blocks includes a storage start address and a plurality of rows of data;

[0007] Parsing the at least two sub-datasets in parallel;

[0008] The step of parsing the at least two sub-data sets in parallel includes:

[0009] In response to the division position of the data set, a target data block among the multiple data blocks is divided into a first sub-data set and a second sub-data set, and the target data block is divided into a first part and a second part, and the first sub-data set and the second sub-data set are parsed in parallel; the first part includes a storage start address of the target data block;

[0010] In response to the first sub-dataset and the second sub-dataset being parsed, a storage start address corresponding to the second part is determined according to a storage start address and an end address corresponding to the first part obtained by parsing.

[0011] Optionally, determining the storage start address corresponding to the second part according to the storage start address and end address corresponding to the first part obtained by parsing includes:

[0012] Determining a preset starting address of the second part according to a storage starting address corresponding to the first part;

[0013] A storage start address corresponding to the second part is determined according to an end address corresponding to the first part and a preset start address of the second part.

[0014] Optionally, the parsing the first sub-dataset and the second sub-dataset in parallel includes:

[0015] Acquire multiple rows of data in the first sub-dataset and the second sub-dataset;

[0016] The multiple rows of data in the first sub-dataset and the second sub-dataset are converted into binary data.

[0017] Optionally, the validity of the storage start address corresponding to each of the data blocks is verified.

[0018] Optionally, after the at least two sub-datasets are parsed in parallel, the method further comprises:

[0019] The binary data is written into the memory according to the storage start address corresponding to each of the data blocks.

[0020] Optionally, before dividing the data set into at least two sub-data sets, the method further comprises:

[0021] The numerical file is read and preprocessed to obtain the data set.

[0022] Optionally, the preprocessing includes removing duplicate and invalid data blocks.

[0023] As a second aspect of the present application, a data processing device is provided, comprising: a division module and a parsing module;

[0024] The partitioning module is used to partition a data set into at least two sub-data sets; the data set includes a plurality of data blocks, each of which includes a storage start address and a plurality of rows of data;

[0025] The parsing module is used to parse the at least two sub-data sets in parallel;

[0026] Wherein, the parsing module is specifically used for:

[0027] In response to the division position of the data set, a target data block among the multiple data blocks is divided into a first sub-data set and a second sub-data set, and the target data block is divided into a first part and a second part, and the first sub-data set and the second sub-data set are parsed in parallel; the first part includes a storage start address of the target data block;

[0028] In response to the first sub-dataset and the second sub-dataset being parsed, a storage start address corresponding to the second part is determined according to a storage start address and an end address corresponding to the first part obtained by parsing.

[0029] Optionally, the parsing module is further configured to determine a preset starting address of the second part according to a storage starting address corresponding to the first part;

[0030] A storage start address corresponding to the second part is determined according to an end address corresponding to the first part and a preset start address of the second part.

[0031] Optionally, the parsing module is further configured to obtain a plurality of rows of data in the first sub-dataset and the second sub-dataset;

[0032] The multiple rows of data in the first sub-dataset and the second sub-dataset are converted into binary data.

[0033] Optionally, the data processing device further includes a verification module;

[0034] The verification module is used to verify the validity of the storage start address corresponding to each of the data blocks.

[0035] Optionally, the data processing device further includes a writing module;

[0036] The writing module is used to write the binary data into the memory according to the storage start address corresponding to each of the data blocks.

[0037] Optionally, the data processing device further includes a processing module;

[0038] The processing module is used to read the numerical file and perform preprocessing to obtain the data set.

[0039] Optionally, the preprocessing includes removing duplicate and invalid data blocks.

[0040] As a third aspect of the present application, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data processing method when executing the program.

[0041] As a fourth aspect of the present application, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned data processing method provided in the present application.

[0042] As can be seen from the above, the data processing method, device, equipment and storage medium provided by the present application can significantly improve the speed of data processing by dividing the data set into at least two sub-data sets and performing parallel parsing. Parallel processing can take advantage of the advantages of multi-core processors and reduce the overall data processing time. Furthermore, by dynamically adjusting the number of sub-data sets, it can be optimized according to the size of the data set and the availability of computing resources, making the data processing method more flexible and adaptable to different computing environments. Further, in the parallel parsing process, even if the target data block is divided into a first part and a second part, and is located in different sub-data sets, the method can still use the storage start address contained in the first part to determine the storage start address of the second part. This helps to ensure the integrity and continuity of the data, avoid data loss or damage during data partitioning and parallel processing, thereby enhancing the performance when processing large-scale data sets, significantly improving the speed and efficiency of data processing, while maintaining the integrity and accuracy of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 A schematic diagram of a memory array provided in an embodiment of the present application;

[0045] Figure 2 A schematic diagram of a readmemh file provided in an embodiment of the present application;

[0046] Figure 3 Another schematic diagram of a readmemh file provided in an embodiment of the present application;

[0047] Figure 4 A schematic diagram of the correspondence between data and memory in a readmemh file provided in an embodiment of the present application;

[0048] Figure 5 A flowchart of a data processing method provided in an embodiment of the present application;

[0049] Figure 6 A flowchart of another data processing method provided in an embodiment of the present application;

[0050] Figure 7 A flowchart of another data processing method provided in an embodiment of the present application;

[0051] Figure 8 A flowchart of another data processing method provided in an embodiment of the present application;

[0052] Fig. 9 A flowchart of another data processing method provided in an embodiment of the present application;

[0053] Fig.10 A flowchart of another data processing method provided in an embodiment of the present application;

[0054] Fig.11 A flowchart of another data processing method provided in an embodiment of the present application;

[0055] Fig.12 A flowchart of another data processing method provided in an embodiment of the present application;

[0056] Fig.13 A schematic diagram of the composition of a data processing device provided in an embodiment of the present application;

[0057] Fig.14 A schematic diagram of the composition of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0059] It should be noted that, in the embodiments of the present application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a concrete way. Unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should have the usual meaning understood by people with ordinary skills in the field to which the present application belongs. SUMMARY OF THE INVENTION

[0061] In the related art, in high-performance chips, such as system on chip (SoC), central processing unit (CPU) and graphics processing unit (GPU), memory is crucial to its performance. Figure 1 As shown, the register transfer level (RTL) describes a very large memory array with a storage size of 2 31 entries, each entry is 64 bits (8 bytes), and a total of 16GB of storage space is required. When simulating, the user needs to provide a numerical file for the memory, and use the hardware simulator to write the value of the numerical file into the memory to initialize the memory. The readmemh file is a commonly used memory file format, and its format is as follows Figure 2 As shown, the format uses @+address to describe the starting address of a memory. The legal range of the address is [0, the maximum address of the memory]. The address is followed by the data to be written to the address. For example, Figure 3 As shown, the value of the memory row at address [0, 5] is assigned to: 0123456789abcdefg, the value of the memory row at address [511, 516] is assigned to: 0123456789abcdefg, and the value of the memory row at address [767, 772] is assigned to: 0123456789abcdefg. Correspondingly, the corresponding relationship between the data and memory of the readmemh file is as follows Figure 4 As shown, it can be seen that for large-scale memory, the corresponding text file will also be very large. How to efficiently parse such a huge file is crucial to the memory writing speed of the hardware emulator.

[0062] The inventors of the present application have found that conventional serial parsing methods, such as Figure 5 As shown in the figure, first, read all memory files into the memory, then parse the memory byte by byte, convert the data in the memory into the data structure required by the underlying layer, that is, convert the hexadecimal string into binary data, and finally, call the underlying interface to write the binary data into the memory. This serial step is the simplest way, and its time consumption will be the sum of the time of the three steps, but this method cannot give full play to the performance advantages of multi-core CPUs. Therefore, it can be optimized through pipeline. When reading memory files in the traditional way, all files will be read into the memory, which will cause the parsing work to wait until the file is completely read before it can start, such as Figure 6As shown in the figure, after adopting the pipeline, ideally, the file reading will adopt the method of reading a fixed-size data block each time, and after reading, it will be sent to the parsing thread for data parsing, and then the block of data will be written. However, in actual conditions, the speed of each step in the pipeline is not equal, and the speed of the pipeline will depend on the slowest step. In this task, if only the pipeline method is used for optimization, the speed still cannot reach the ideal value, and the bottleneck will be in parsing the data. Therefore, the method of improving the speed of parsing data is also very important. In addition, since the throughput of the system is limited, it is necessary to reduce the time of parsing data as much as possible so that the throughput of the system can be close to the upper limit of the hardware speed.

[0063] In order to solve the above problems, the present application provides a data processing method, which can significantly improve the speed of data processing by dividing a data set into at least two sub-data sets and performing parallel parsing. Parallel processing can take advantage of multi-core processors and reduce the overall data processing time. Furthermore, by dynamically adjusting the number of sub-data sets, it can be optimized according to the size of the data set and the availability of computing resources, making the data processing method more flexible and adaptable to different computing environments. Further, in the parallel parsing process, even if the target data block is divided into a first part and a second part, and is located in different sub-data sets, the method can still use the storage start address contained in the first part to determine the storage start address of the second part. This helps to ensure the integrity and continuity of the data, avoid data loss or damage during data partitioning and parallel processing, thereby enhancing the performance when processing large-scale data sets, significantly improving the speed and efficiency of data processing, while maintaining the integrity and accuracy of the data.

[0064] After introducing the basic principles of the present application, various non-limiting implementation methods of the present application are described in detail below.

[0065] Figure 7 A flow chart of a data processing method provided in an embodiment of the present application. Figure 7 As shown, the data processing method provided by the present application specifically includes the following steps:

[0066] S101. Divide a data set into at least two sub-data sets.

[0067] Among them, the data set contains multiple data blocks, such as the above Figure 2 As described above, each data block includes a storage start address and multiple lines of data.

[0068] In some embodiments, the number of sub-data sets is dynamically determined based on the size or number of data blocks. For example, a load balancing algorithm is used to dynamically adjust the allocation of sub-data sets based on the current system load, and by dynamically dividing the data set into multiple sub-data sets, each sub-data set is ensured to be relatively balanced in size, complexity, or computing requirements.

[0069] In some embodiments, Figure 8 As shown, before S101, the data processing method provided in the embodiment of the present application further includes the following S100:

[0070] S100, reading a numerical file and performing preprocessing to obtain a data set.

[0071] In some embodiments, a standard file I / O operation is used to read a numerical file, or a memory-mapped file technique is used to map the file directly in memory to improve reading efficiency. Further, according to the file format, the corresponding parsing library is used to convert the data into a structured data set for easy processing, for example, converting a CSV into a database table. Then, invalid data blocks are identified and removed according to preset rules (such as null values, illegal characters, and data that does not conform to the format). The preprocessed data is stored in memory as an input data set for subsequent steps.

[0072] In some embodiments, preprocessing includes removing duplicate and invalid data blocks, and the preprocessing method includes: identifying and removing duplicate data blocks through regular expressions; using the deduplication function of the database or data framework to clean the data; using a hash table to quickly identify and remove duplicate items, etc.

[0073] S102. Analyze at least two sub-datasets in parallel.

[0074] In some embodiments, a thread pool is created based on the hardware resources of the system and the number of sub-datasets, each sub-dataset is assigned to a thread in the thread pool for parsing, and the threads in the thread pool parse the assigned sub-datasets in parallel.

[0075] In some embodiments, a cluster comprising multiple nodes is configured in a distributed computing environment, and sub-data sets are distributed to different nodes in the cluster to utilize the parallel computing capabilities of the cluster. On each node, an independent parsing task is started to parse the assigned sub-data set, and the intermediate results are transmitted over the network.

[0076] In some embodiments, Figure 7 As shown, S102 can be specifically implemented as follows S1021-S1022:

[0077] S1021. Divide a target data block among the multiple data blocks into a first sub-data set and a second sub-data set in response to a division position of the data set, and the target data block is divided into a first part and a second part, and the first sub-data set and the second sub-data set are parsed in parallel.

[0078] The first part includes the storage start address of the target data block.

[0079] In some embodiments, the division position is determined based on the size, content characteristics, byte offset or preset segmentation rules of the data block. The division position can be as follows: Fig. 9 As shown in the figure, the data block is complete after the line break at the end of the data block or before the head address. Fig.11 As shown, the target data block is divided into a first part and a second part at the division position. Then, two parallel tasks are created by multithreading technology to parse the first sub-data set and the second sub-data set at the same time, which significantly improves the speed of data processing.

[0080] In some embodiments, parallel parsing includes: obtaining multiple lines of data in the first sub-dataset and the second sub-dataset; converting the multiple lines of data in the first sub-dataset and the second sub-dataset into binary data, for example, using a function or library provided by a programming language to convert hexadecimal data in string form into binary format.

[0081] S1022: In response to the first sub-data set and the second sub-data set being parsed, determine a storage start address corresponding to the second part according to a storage start address and an end address corresponding to the first part obtained by parsing.

[0082] In some embodiments, Fig.10 As shown, after parsing the first sub-data set and the second sub-data set, the storage start address corresponding to each data block can be obtained, for example, Fig.11 The clear address (Clear Addr) corresponding to the first data block (chunk data 1) in the figure. According to the storage start address corresponding to the first part, the preset start address of the second part is determined, for example, according to the storage start address and data block size of the first part, or using a fixed offset or algorithm to assign a placeholder or initial estimate associated with the first part to the second part, that is, the fuzzy address (Fuzzy Addr) of the second data block (chunkdata2) in the figure; according to the end address corresponding to the first part and the preset start address of the second part, the storage start address corresponding to the second part is determined, and at this time, the storage start address corresponding to the second part is the sum of the fuzzy Addr and the end address (endAddr of chunk data1) corresponding to the first part.

[0083] The data processing method provided by the present application can significantly improve the speed of data processing by dividing a data set into at least two sub-data sets and performing parallel parsing. Parallel processing can take advantage of multi-core processors and reduce the overall data processing time. Furthermore, by dynamically adjusting the number of sub-data sets, it can be optimized according to the size of the data set and the availability of computing resources, making the data processing method more flexible and adaptable to different computing environments. Further, in the parallel parsing process, even if the target data block is divided into a first part and a second part, and is located in different sub-data sets, the method can still use the storage start address contained in the first part to determine the storage start address of the second part. This helps to ensure the integrity and continuity of the data, avoid data loss or damage during data partitioning and parallel processing, thereby enhancing the performance when processing large-scale data sets, significantly improving the speed and efficiency of data processing, while maintaining the integrity and accuracy of the data.

[0084] In some embodiments, Fig.11 As shown, the data processing method provided in the embodiment of the present application further includes the following S103:

[0085] S103: Verify the validity of the storage start address corresponding to each data block.

[0086] In some embodiments, the verification method includes: checking whether the storage start address of each data block is within the valid address range of the memory, verifying whether the address exceeds the physical or logical boundary of the memory; and using a data structure (such as a hash table) to record the allocated addresses, checking whether the newly allocated addresses conflict with the existing addresses, and ensuring that no overlapping address areas are allocated to different data blocks; in addition, verifying whether the storage start address is consistent with the metadata of the data block (such as size, type, attributes, etc.), ensuring that the address information in the metadata matches the actual allocated address, etc. In the case where the storage start address is found to be invalid, an error is recorded and corresponding error handling measures are taken (such as reallocating the address, terminating the operation, etc.).

[0087] In some embodiments, Fig.12 As shown, the data processing method provided in the embodiment of the present application further includes the following S104:

[0088] S104. Write the binary data into the memory according to the storage start address corresponding to each data block.

[0089] In some embodiments, the data write interface is called based on the storage start address and binary data obtained by parallel parsing. At this time, the storage start address can be used to locate the specific storage unit that needs to be written, and then, based on the storage start address, the binary data is written to the corresponding position of the memory.

[0090] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0091] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0092] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a data processing device.

[0093] refer to Fig.13 , the data processing device comprises: a division module 1301 and a parsing module 1302;

[0094] The division module 1301 is used to divide a data set into at least two sub-data sets; the data set includes a plurality of data blocks, each of which includes a storage start address and a plurality of rows of data;

[0095] The parsing module 1302 is used to parse the at least two sub-data sets in parallel;

[0096] The analysis module 1302 is specifically used for:

[0097] In response to the division position of the data set, a target data block among the multiple data blocks is divided into a first sub-data set and a second sub-data set, and the target data block is divided into a first part and a second part, and the first sub-data set and the second sub-data set are parsed in parallel; the first part includes a storage start address of the target data block;

[0098] In response to the first sub-dataset and the second sub-dataset being parsed, a storage start address corresponding to the second part is determined according to a storage start address and an end address corresponding to the first part obtained by parsing.

[0099] In some embodiments, the parsing module 1302 is further configured to determine a preset starting address of the second part according to a storage starting address corresponding to the first part;

[0100] A storage start address corresponding to the second part is determined according to an end address corresponding to the first part and a preset start address of the second part.

[0101] In some embodiments, the parsing module 1302 is further configured to obtain multiple rows of data in the first sub-dataset and the second sub-dataset;

[0102] The multiple rows of data in the first sub-dataset and the second sub-dataset are converted into binary data.

[0103] In some embodiments, the data processing device further includes a verification module 1303;

[0104] The verification module 1303 is used to verify the validity of the storage start address corresponding to each of the data blocks.

[0105] In some embodiments, the data processing device further includes a writing module 1304;

[0106] The writing module 1304 is used to write the binary data into the memory according to the storage start address corresponding to each data block.

[0107] In some embodiments, the data processing device further includes a processing module 1305;

[0108] The processing module 1305 is used to read the numerical file and perform preprocessing to obtain the data set.

[0109] In some embodiments, the pre-processing includes removing duplicate and invalid data blocks.

[0110] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0111] The device of the above embodiment is used to implement the corresponding data processing method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0112] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data processing method described in any of the above embodiments is implemented.

[0113] Fig.14A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1060. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other in communication within the device through the bus 1060.

[0114] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0115] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0116] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0117] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0118] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0119] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0120] The electronic device of the above embodiment is used to implement the corresponding data processing method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0121] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the data processing method described in any of the above embodiments.

[0122] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0123] The computer instructions stored in the storage medium of the above embodiments are used to enable the computer to execute the data processing method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0124] Based on the same inventive concept, corresponding to the data processing method described in any of the above embodiments, the present disclosure also provides a computer program product, which includes a computer program. In some embodiments, the computer program is executable by one or more processors so that the processors execute the data processing method. Corresponding to the execution subject corresponding to each step in each embodiment of the data processing method, the processor that executes the corresponding step may belong to the corresponding execution subject.

[0125] The computer program product of the above embodiment is used to enable the processor to execute the data processing method described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0126] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0127] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0128] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0129] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Dividing a data set into at least two sub-data sets; the data set includes a plurality of data blocks, each of the data blocks includes a storage start address and a plurality of rows of data; Parsing the at least two sub-datasets in parallel; The step of parsing the at least two sub-data sets in parallel includes: In response to the division position of the data set, a target data block among the multiple data blocks is divided into a first sub-data set and a second sub-data set, and the target data block is divided into a first part and a second part, and the first sub-data set and the second sub-data set are parsed in parallel; the first part includes a storage start address of the target data block; In response to the first sub-dataset and the second sub-dataset being parsed, a storage start address corresponding to the second part is determined according to a storage start address and an end address corresponding to the first part obtained by parsing.

2. The method according to claim 1, characterized in that The determining, according to the storage start address and the end address corresponding to the first part obtained by parsing, the storage start address corresponding to the second part comprises: Determining a preset starting address of the second part according to a storage starting address corresponding to the first part; A storage start address corresponding to the second part is determined according to an end address corresponding to the first part and a preset start address of the second part.

3. The method according to claim 1, characterized in that The step of parsing the first sub-dataset and the second sub-dataset in parallel includes: Acquire multiple rows of data in the first sub-dataset and the second sub-dataset; The multiple rows of data in the first sub-dataset and the second sub-dataset are converted into binary data.

4. The method according to claim 3, characterized in that After the parallel parsing of the at least two sub-data sets, the method further comprises: Verify the validity of the storage start address corresponding to each of the data blocks.

5. The method according to claim 4, characterized in that The method further comprises: The binary data is written into the memory according to the storage start address corresponding to each of the data blocks.

6. The method according to claim 1, characterized in that Before dividing the data set into at least two sub-data sets, the method further includes: The numerical file is read and preprocessed to obtain the data set.

7. The method according to claim 6, characterized in that The pre-processing includes removing duplicate and invalid data blocks.

8. A data processing device, characterized in that: The device comprises: a division module and a parsing module; The partitioning module is used to partition a data set into at least two sub-data sets; the data set includes a plurality of data blocks, each of which includes a storage start address and a plurality of rows of data; The parsing module is used to parse the at least two sub-data sets in parallel; Wherein, the parsing module is specifically used for: In response to the division position of the data set, a target data block among the multiple data blocks is divided into a first sub-data set and a second sub-data set, and the target data block is divided into a first part and a second part, and the first sub-data set and the second sub-data set are parsed in parallel; the first part includes a storage start address of the target data block; In response to the first sub-dataset and the second sub-dataset being parsed, a storage start address corresponding to the second part is determined according to a storage start address and an end address corresponding to the first part obtained by parsing.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program. 10 . A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to claim 1 .

Images