A method and apparatus for improving the correlation performance of multi-source heterogeneous data

Abstract

This invention discloses a method and apparatus for improving the performance of multi-source heterogeneous data association. The method includes: Spark performing concurrent partitioned reading of non-file data sources and establishing a mapping relationship between the addresses of physical files on disk and the virtual addresses of the current partition's memory space; writing data from each partition to the current partition's memory space at near-memory speed; the operating system periodically and asynchronously writing data from the current partition's memory space to physical files on disk; reading data using the current partition's memory space plus the physical files on disk, preheating the physical files on disk, and then converting the data into a Spark DataFrame; and automatically adapting the optimal data association method based on the allocated executor memory and the size of the Spark DataFrame. This method and apparatus can improve the performance of data association and automatically adapt to the optimal data association method.

Description

Technical Field

[0001] This invention relates to the field of data association performance, and in particular to a method and apparatus for improving the association performance of multi-source heterogeneous data. Background Technology

[0002] When using Spark to connect heterogeneous data sources, if the data source is a non-file data source, such as MySQL or Elasticsearch, during the data connection process, each partition in Spark needs to connect to the data source such as MySQL and Elasticsearch. Each partition needs to pull the full amount of data. When there are too many partitions, a large number of connections need to be created and a large amount of data needs to be pulled, resulting in very low connection efficiency.

[0003] If you use Spark to extract non-file data sources, such as MySQL tables or Elasticsearch indexes, to files in advance, it will generate high disk I / O, consume a lot of time, and affect the performance of data association processing. Summary of the Invention

[0004] To address the aforementioned problems in existing technologies, this invention provides a method and apparatus for improving the performance of multi-source heterogeneous data association. It uses Spark to associate heterogeneous data sources, and when the data source is a non-file data source, it can improve the performance of data association and automatically adapt to the optimal data association method.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] In one embodiment of the present invention, a method for improving the correlation performance of multi-source heterogeneous data is proposed, the method comprising:

[0007] Spark performs concurrent partitioned reads of non-file data sources and establishes a mapping between the addresses of physical files on disk and the virtual addresses of the current partition's memory space.

[0008] Spark writes the data from each partition to the memory space of the current partition at near-memory speed;

[0009] The operating system periodically and asynchronously writes the data in the memory space of the current partition to physical files on the disk;

[0010] Spark reads data using the memory space of the current partition plus the physical file on disk, preheats the physical file on disk, and then converts the data into a Spark DataFrame.

[0011] Spark automatically adapts the optimal data association method based on the allocated executor memory and the size of the Spark DataFrame.

[0012] Furthermore, Spark performs concurrent partitioned reads of non-file data sources and establishes a mapping between the addresses of physical files on disk and the virtual addresses of the current partition's memory space, including:

[0013] Based on the partition field and the number of partitions, read the number of the current partition and read the data of the current partition;

[0014] Create a new physical file on the disk, with the physical file name containing the number of the current partition;

[0015] The program initializes the virtual address of the current partition's memory space, establishing a mapping between the address of the physical file on the disk and the virtual address of the current partition's memory space based on the physical file name on the disk.

[0016] Furthermore, add a file write success flag to the last line of data in the current partition's memory space.

[0017] Furthermore, after the data in the memory space of the current partition is written to the physical file on the disk, a status file is generated on the disk. Based on the data write time in the status file, it is determined whether Spark needs to fetch the non-file data source to each partition to generate files again.

[0018] Furthermore, Spark automatically adapts the optimal data join method based on the allocated executor memory and the size of the Spark DataFrame, including:

[0019] Spark calculates the size of the executor's storage memory based on the ratio of the sum of the executor's execution memory and the executor's storage memory to the total executor memory, as well as the ratio of the executor's execution memory and the sum of the executor's storage memory.

[0020] Calculate the proportion of Spark's DataFrame in the executor's storage memory based on the size of the executor's storage memory and the size of Spark's DataFrame.

[0021] If the proportion of Spark DataFrame's storage memory occupied by the executor is less than a specified threshold, then the Spark DataFrame is sent to each executor and stored in the memory of each executor. When performing data association, the data is read from the current executor.

[0022] If the proportion of storage memory occupied by a Spark DataFrame in the executor exceeds a specified threshold, then when performing data association, data will be read using the memory space of the current partition plus the physical files on the disk.

[0023] In one embodiment of the present invention, an apparatus for improving the correlation performance of multi-source heterogeneous data is also proposed, the apparatus comprising:

[0024] The data writing module is used by Spark to perform concurrent partitioned reading of non-file data sources and establish a mapping relationship between the address of the physical file on the disk and the virtual address of the memory space of the current partition; Spark writes the data of each partition to the memory space of the current partition at a speed close to memory speed; the operating system periodically and asynchronously writes the data of the memory space of the current partition to the physical file on the disk.

[0025] The data reading module is used by Spark to read data using the memory space of the current partition plus the physical files on the disk, preheat the physical files on the disk, and then convert the data into Spark DataFrames.

[0026] The data association adaptation module is used by Spark to automatically adapt the optimal data association method based on the allocated executor memory and the size of the Spark DataFrame.

[0027] Furthermore, Spark performs concurrent partitioned reads of non-file data sources and establishes a mapping between the addresses of physical files on disk and the virtual addresses of the current partition's memory space, including:

[0028] Based on the partition field and the number of partitions, read the number of the current partition and read the data of the current partition;

[0029] Create a new physical file on the disk, with the physical file name containing the number of the current partition;

[0030] The program initializes the virtual address of the current partition's memory space, establishing a mapping between the address of the physical file on the disk and the virtual address of the current partition's memory space based on the physical file name on the disk.

[0031] Furthermore, add a file write success flag to the last line of data in the current partition's memory space.

[0032] Furthermore, after the data in the memory space of the current partition is written to the physical file on the disk, a status file is generated on the disk. Based on the data write time in the status file, it is determined whether Spark needs to fetch the non-file data source to each partition to generate files again.

[0033] Furthermore, Spark automatically adapts the optimal data join method based on the allocated executor memory and the size of the Spark DataFrame, including:

[0034] Spark calculates the size of the executor's storage memory based on the ratio of the sum of the executor's execution memory and the executor's storage memory to the total executor memory, and the ratio of the executor's storage memory to the sum of the executor's execution memory and the executor's storage memory.

[0035] Calculate the proportion of Spark's DataFrame in the executor's storage memory based on the size of the executor's storage memory and the size of Spark's DataFrame.

[0036] If the proportion of Spark DataFrame's storage memory occupied by the executor is less than a specified threshold, then the Spark DataFrame is sent to each executor and stored in the memory of each executor. When performing data association, the data is read from the current executor.

[0037] If the proportion of storage memory occupied by a Spark DataFrame in the executor exceeds a specified threshold, then when performing data association, data will be read using the memory space of the current partition plus the physical files on the disk.

[0038] In one embodiment of the present invention, a computer device is also proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it achieves the aforementioned improvement in the performance of multi-source heterogeneous data association.

[0039] In one embodiment of the present invention, a computer-readable storage medium is also proposed, which stores a computer program that executes to improve the performance of association of multi-source heterogeneous data.

[0040] Beneficial effects:

[0041] 1. This invention supports partitioned reading of non-file data sources, improving the concurrency of data reading.

[0042] 2. This invention enables non-file data sources to be stored in files at near-memory speeds and provides a fault tolerance mechanism.

[0043] 3. The data reading of this invention can make full use of the leaf cache, greatly improving the performance of data reading.

[0044] 4. During the data validity period, this invention can avoid repeatedly fetching data when the application runs multiple times, thereby improving performance.

[0045] 5. This invention uses Spark to automatically adapt the optimal data association method based on the allocated executor memory and the size of the data. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the method for improving the correlation performance of multi-source heterogeneous data according to the present invention;

[0047] Figure 2 This is a schematic diagram of memory allocation for the Spark Executor Memory of this invention;

[0048] Figure 3 This is a schematic diagram of the device structure for improving the correlation performance of multi-source heterogeneous data according to the present invention;

[0049] Figure 4 This is a schematic diagram of the computer device structure of the present invention. Detailed Implementation

[0050] The principles and spirit of the present invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0051] Those skilled in the art will recognize that embodiments of the present invention can be implemented as an apparatus, device, node, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0052] This invention proposes a method and apparatus to improve the performance of multi-source heterogeneous data association. It uses Spark to associate heterogeneous data sources. When the data source is a non-file data source, Spark performs concurrent partitioned reading of the non-file data source and establishes a mapping relationship between the addresses of physical files on the disk and the virtual addresses of the current partition's memory space. Spark writes the data from each partition to the current partition's memory space at near-memory speed. The operating system periodically and asynchronously writes the data from the current partition's memory space to the physical files on the disk. To avoid crashes caused by the current partition's memory space not being written to the disk in a timely manner, a fault tolerance mechanism is provided to ensure data recovery from failures. Based on the data write time in the status file on the disk, it is determined whether Spark needs to re-fetch the non-file data source to each partition to generate files. If the data is within its validity period, repeated data fetching can be avoided during multiple application runs, thereby improving performance. Spark reads data using the current partition's memory space plus the physical files on the disk, and preheats the physical files on the disk, storing as much data as possible in the current partition's memory space to improve data reading efficiency. The data is then converted into a Spark DataFrame. Spark determines whether to send the Spark DataFrame to each executor based on the allocated executor memory and the size of the Spark DataFrame, thereby improving performance and automatically adapting to the optimal data association method.

[0053] The principles and spirit of the present invention will be explained in detail below with reference to several representative embodiments.

[0054] Figure 1 This is a schematic diagram of the method for improving the correlation performance of multi-source heterogeneous data according to the present invention. Figure 1 As shown, the method includes:

[0055] 1. Customize Spark's interface for reading non-file data sources.

[0056] By using Spark's non-file data source reading interface, the underlying implementation details are hidden, greatly simplifying the process of reading data for users.

[0057] Create a new data source class that inherits from the abstract classes RelationProvider and DataSourceRegister, and implement the relevant methods as follows:

[0058]

[0059] When reading data, you only need to specify the data source name. For example:

[0060] spark.read.options(map)).format("mysql").load()

[0061] The parameters needed to read the data source, such as the MySQL host IP address, port, partition field, and number of partitions, can be specified using variables of the map data type. For example, the partition field can be set to `partition_field`, and the number of partitions to `partition_size`. The specification method is as follows:

[0062] val map=Map("partition_field"->"field","partition_size"->10).

[0063] 2. Internal implementation of interfaces for non-file data sources

[0064] When reading data, two parameters need to be specified: (1) partition field, such as the name of a field in MySQL; (2) number of partitions, which can be an optional parameter, such as the default value of 10.

[0065] For clarity, we assume the partition field is partition_field and the number of partitions is partition_size.

[0066] Spark creates a number of partitions equal to the partition_size. Within each partition, data from a non-file data source is read. The entire dataset is distributed across each partition and read separately. The data read from all partitions is then merged together to form a single full dataset.

[0067] Each partition has a unique number, starting from 0. The logic for reading data from each partition is as follows:

[0068] (1) Determine the range of data to be read

[0069] The query conditions for the current partition are constructed based on the partition field partition_field and the number of partitions partition_size. The number of the current partition is read and represented by partition_id.

[0070] The logic for constructing the query conditions for the current partition is as follows: (a) Perform a hash calculation on the partition field to generate a numeric hash value. (b) Take the remainder of the generated hash value with respect to the number of partitions. (3) The remainder is the number of the current partition. The formula is as follows: hash(partition_field) % partition_size = partition_id.

[0071] If it is a MySQL data source, the query statement for the current partition would be similar to: select * from table wherehash(partition_field) % partition_size = partition_id.

[0072] After the above operations, it can be ensured that the data read from each partition does not overlap, and the data from all partitions is merged together to read a complete set of data.

[0073] (2) Data in each partition is written to the disk at near memory speed.

[0074] The traditional data writing process involves two steps: first, data is written to the memory space of the current program process; then, the data in memory is copied to the operating system kernel's I / O buffer; and finally, it is written to the disk file. This process involves two data copies: one from memory to the operating system kernel's I / O buffer, and the other from the operating system kernel's I / O buffer to the disk file. Copying data from memory to the operating system kernel's I / O buffer is particularly CPU and memory intensive, impacting file writing performance.

[0075] To solve the above problems, this invention writes the data of each partition to the disk at a speed close to that of memory.

[0076] (a) Create a new physical file on the disk of the host where the Spark task's Executor process is located. The physical file name contains the partition identifier. For example, add the partition number as a suffix to the physical file name (e.g., part_0.csv, where 0 is the partition number, i.e., the partition identifier, and csv is the file format). This partition number is used to identify the data source of the file.

[0077] (b) Initialize the virtual address of the memory space of the current partition of the program process. Using the physical file name of the current partition, use the Java built-in MappedByteBuffer.map function to establish the mapping relationship between the address of the physical file on the disk and the virtual address of the memory space of the current partition.

[0078] (c) Read the data from the current partition and write it to the memory space of the current partition, i.e., the leaf cache. Any unallocated memory in the Linux operating system can be used for the leaf cache. Because the data write operation to the current partition is performed in memory, the data writing efficiency is very fast.

[0079] (d) Add a file write success flag to the last line of the current partition's data. For example, write the string "_SUCCESS". Data with the file write success flag is also first written to the leaf cache of the current partition.

[0080] The purpose of this step is because leaf caches may be at risk of data loss. If the data in the leaf cache is not written to the physical files on disk in a timely manner, and the host crashes, it may lead to data loss. Therefore, a fault tolerance mechanism is needed to recover data from failures.

[0081] (e) The Linux operating system periodically and asynchronously writes data from the leaf cache to the physical files on the disk. However, if the operating system fails to write the data from the leaf cache to the physical files on the disk in a timely manner, the program on the current partition will exit abnormally. When the program restarts, it checks the last line of data in each partition. Since it only checks the last line, the process is very fast. If the last line does not contain a file write success flag, it means that the data written from the leaf cache to the physical files on the disk by the operating system last time was incomplete, i.e., it failed. In this case, the data write process for the current partition is restarted, the dirty data in the current partition is deleted, and the data in the current partition is rewritten to ensure data consistency.

[0082] 3. Generate status file

[0083] After data in the leaf cache is written to a physical file on disk, a status file is generated on disk to record the start time of the current data write to the leaf cache. For example, the file format is as follows:

[0084] {"lastUpdatedTime":"2022-10-2912:34:46"}

[0085] The purpose of recording this time is that some non-file data sources update infrequently, for example, perhaps only once a day. When a Spark program re-executes a data association task, it compares the current time with the time in the state file to determine whether to re-fetch the data and update the file. The data update frequency can be passed in as a parameter (the parameter value is 1 hour or 10 minutes, etc.), and Spark uses this parameter to determine whether to re-fetch the data and update the file.

[0086] 4. Spark reads files and converts them into Spark DataFrames.

[0087] (1) Determine whether data needs to be fetched again;

[0088] During data reading, a parameter specifying the data update frequency is passed in. Based on the parameter's value, the current time, and the time in the status file, it is determined whether data needs to be fetched to the file again. The rules are as follows: (a) If the time in the status file plus the data update frequency parameter value (e.g., 1 hour or 10 minutes) is less than or equal to the current time, it means the data has not expired, and there is no need to fetch data from the non-file data source again. (b) Conversely, if the data from the non-file data source is not expired, then it is necessary to rewrite and fetch the data from the non-file data source, updating the file.

[0089] (2) Read the files generated in each partition using Spark and convert the files into Spark DataFrames;

[0090] The traditional data reading process involves: (a) reading data from the disk and storing it in the operating system's kernel I / O buffer; and (b) reading the data from the operating system's kernel I / O buffer into the current program's memory space. This process involves two data copying operations: reading data into the operating system's kernel I / O buffer and reading data from the operating system's kernel I / O buffer into the current program's memory space. The latter operation is particularly resource-intensive, consuming significant CPU and memory resources and impacting read performance.

[0091] To resolve the data reading issue mentioned above, the following data reading process will be executed:

[0092] (a) Traverse the physical files of each partition on the disk and map the physical files of each partition to a virtual address in a memory space.

[0093] (b) When reading data, first check if the corresponding data is contained in the leaf cache. If the data is contained, read it directly from the leaf cache. If the corresponding data is not contained in the leaf cache, load the data from the physical file on the disk into the leaf cache and read it again.

[0094] (c) In order to improve the efficiency of data reading, the physical files of each partition on the disk are preheated periodically, and as much data as possible from the physical files of each partition on the disk is loaded into the leaf cache.

[0095] (d) During reading, filter out the last line of each file (the line that indicates that the file was written successfully) and convert the data into a Spark DataFrame (a distributed dataset in Spark, similar to a distributed table, which contains data and information such as the field names of the data).

[0096] 5. Determine the data association strategy based on the data volume.

[0097] The memory allocation for Spark Executor Memory is as follows: Figure 2 As shown.

[0098] The memory of a Spark executor consists of three parts: Execution memory, Storage memory, and other memory (memory other than Execution and Storage).

[0099] The memory of the Spark executor is the memory specified when submitting the Spark application. It is determined whether to send the Spark DataFrame generated in the previous step to each executor or read the Spark DataFrame generated in the previous step into a task corresponding to each partition, based on the size of the Spark DataFrame generated in the previous step (Spark DataFrame comes from disk, and the file size is calculated when reading data from disk) and the memory of the executor.

[0100] If the Spark DataFrame generated in the previous step has a small amount of data, you can consider storing a complete copy of the Spark DataFrame in each executor. When each task performs data association, it does not need to pull data from the disk or other nodes, but only needs to read data from the current executor, which greatly improves the efficiency of data association.

[0101] If the Spark DataFrame generated in the previous step is large, and the memory of each executor is insufficient to store a full copy of the data, then reading the data using the leaf cache and physical files on disk from the previous step can significantly improve performance. Conversely, directly joining data from the database would require fetching a full copy of the database for each partition, resulting in substantial network and disk I / O overhead.

[0102] The logic for determining whether to send the Spark DataFrame generated in the previous step to each executor is as follows:

[0103] Spark has two parameters that control the amount of memory used, and all Spark parameters can be specified when submitting a task.

[0104] The first parameter, spark.memory.fraction, represents the percentage of total executor memory that is the sum of Executor and Storage memory. The default value is 60%. For example, if the executor memory is 10GB, then the sum of Executor and Storage memory is 6GB by default.

[0105] The second parameter, spark.memory.storageFraction, represents the proportion of Storage memory to the sum of Executor and Storage memory. The default value is 0.5, which means that Executor and Storage memory each account for half.

[0106] After generating the Spark DataFrame in the previous step, the data size is calculated, and then the Storage memory size is calculated based on the two parameters mentioned above. The Spark DataFrame size is then compared to the Storage memory size to calculate the proportion of Storage memory occupied by the Spark DataFrame from the previous step. If the proportion is less than a specified threshold (e.g., 0.2 is an optimized proportion), the Spark DataFrame from the previous step is sent to each executor. In this step, the value of the parameter `spark.sql.autoBroadcastJoinThreshold` is set to the size of the Spark DataFrame from the previous step. When performing data joins, Spark actively sends the entire data of the Spark DataFrame from the previous step to each executor, storing it in the memory of each executor. During data joins, there is no data shuffle (shuffle is a Spark-specific concept that refers to data being transferred over the network between nodes; excessive shuffle generates a large amount of disk I / O and network I / O, which can impact performance; the optimization here mainly aims to eliminate shuffle).

[0107] If the proportion is greater than the specified threshold, no processing will be performed on the DataFrame during data association. Instead, the DataFrame will be directly associated with the data using the leaf cache + physical file on disk method described earlier.

[0108] After the above process is completed, Spark will automatically adapt the optimal data association method based on the allocated executor memory and the size of the Spark DataFrame.

[0109] It should be noted that although the operation of the method of the present invention has been described in a specific order in the above embodiments and figures, this does not require or imply that the operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0110] To provide a clearer explanation of the above-mentioned method for improving the correlation performance of multi-source heterogeneous data, a specific embodiment is described below. However, it is worth noting that this embodiment is only for better illustrating the present invention and does not constitute an improper limitation of the present invention.

[0111] Example:

[0112] 1. Import the data source package

[0113] / / Note: The import statement is used to import the package. This package is defined in "1. Customizing Spark's interface for reading non-file data sources". .* indicates that all classes under this package are imported.

[0114] import org.ds.multi.datasource.*

[0115] 2. Define the parameters of the data source

[0116] / / Define a collection of data type Map, named optionsMap, and define the data source connection information and number of partitions, etc.

[0117]

[0118]

[0119] 3. Load the data source using the defined parameters.

[0120] / / df is the variable name of the loaded dataframe. The data source name is mysql_ds, which was defined using shortName in "1. Customizing Spark's Interface for Reading Non-File Data Sources". The optionsMap parameter defined in the previous step is specified through options.

[0121] val df=spark.read.format("mysql_ds").options(optionsMap).load()

[0122] 4. Registry

[0123] With the DataFrame generated in the previous step, you can use the returned DataFrame to register a temporary table, and then write any SQL, including SQL for data relationships.

[0124] Registry methods:

[0125] df.createOrReplaceTempView("temp_table").

[0126] The table name was specified as temp_table above.

[0127] Based on the same inventive concept, this invention also proposes an apparatus for improving the correlation performance of multi-source heterogeneous data. The implementation of this apparatus can refer to the implementation of the method described above, and repeated details will not be elaborated further. The term "module" used below can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0128] Figure 3 This is a schematic diagram of the device structure for improving the correlation performance of multi-source heterogeneous data according to the present invention. (See diagram below.) Figure 3 As shown, the device includes:

[0129] The data writing module 101 is used by Spark to perform concurrent partitioned reading of non-file data sources and establish a mapping relationship between the addresses of physical files on disk and the virtual addresses of the current partition's memory space; specifically as follows:

[0130] Based on the partition field and the number of partitions, read the number of the current partition and read the data of the current partition;

[0131] Create a new physical file on the disk, with the physical file name containing the number of the current partition;

[0132] The program initializes the virtual address of the current partition's memory space, establishing a mapping between the address of the physical file on the disk and the virtual address of the current partition's memory space based on the physical file name on the disk.

[0133] Spark writes the data from each partition to the current partition's memory space at near-memory speed. A success message is added to the last line of the data in the current partition's memory space.

[0134] The operating system periodically and asynchronously writes the data in the memory space of the current partition to physical files on the disk.

[0135] After the data in the memory space of the current partition is written to the physical file on the disk, a status file is generated on the disk. Based on the data write time in the status file, it is determined whether Spark needs to fetch the non-file data source to each partition to generate files again.

[0136] The data reading module 102 is used by Spark to read data using the memory space of the current partition plus the physical files on the disk, preheat the physical files on the disk, and then convert the data into a Spark DataFrame.

[0137] The data association adaptation module 103 is used by Spark to automatically adapt the optimal data association method based on the allocated executor memory and the size of the Spark DataFrame; specifically as follows:

[0138] Spark calculates the size of the executor's storage memory based on the ratio of the sum of the executor's execution memory and the executor's storage memory to the total executor memory, as well as the ratio of the executor's execution memory and the sum of the executor's storage memory.

[0139] Calculate the proportion of Spark's DataFrame in the executor's storage memory based on the size of the executor's storage memory and the size of Spark's DataFrame.

[0140] If the proportion of Spark DataFrame's storage memory occupied by the executor is less than a specified threshold, then the Spark DataFrame is sent to each executor and stored in the memory of each executor. When performing data association, the data is read from the current executor.

[0141] If the proportion of storage memory occupied by a Spark DataFrame in the executor exceeds a specified threshold, then when performing data association, data will be read using the memory space of the current partition plus the physical files on the disk.

[0142] It should be noted that although several modules of the apparatus for improving the correlation performance of multi-source heterogeneous data are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided and embodied by multiple modules.

[0143] Based on the aforementioned inventive concept, such as Figure 4 As shown, the present invention also proposes a computer device 200, including a memory 210, a processor 220, and a computer program 230 stored in the memory 210 and executable on the processor 220. When the processor 220 executes the computer program 230, it implements the aforementioned method for improving the correlation performance of multi-source heterogeneous data.

[0144] Based on the aforementioned inventive concept, the present invention also proposes a computer-readable storage medium storing a computer program that executes the aforementioned performance improvement of multi-source heterogeneous data association.

[0145] The method and apparatus proposed in this invention for improving the performance of multi-source heterogeneous data association support partitioned reading of non-file data sources, thereby increasing the concurrency of data reading; non-file data sources can be stored as physical files on disk at near-memory speeds and fault tolerance mechanisms are provided; data reading can fully utilize leaf caches, greatly improving data reading performance; if the data is within its validity period, it can avoid repeatedly fetching data from non-file data sources during multiple application runs, thus improving performance; Spark will automatically adapt the optimal data association method based on the allocated executor memory and the size of the Spark DataFrame.

[0146] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

[0147] Regarding the limitation of the scope of protection of this invention, those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solution of this invention are still within the scope of protection of this invention.

Claims

1. A method for improving the correlation performance of multi-source heterogeneous data, characterized in that, The method includes: Spark performs concurrent partitioned reads of non-file data sources and establishes a mapping between the addresses of physical files on disk and the virtual addresses of the current partition's memory space, including: Based on the partition field and the number of partitions, read the number of the current partition and read the data of the current partition; Create a new physical file on the disk, with the physical file name containing the number of the current partition; The program initializes the virtual address of the current partition's memory space and establishes a mapping between the address of the physical file on the disk and the virtual address of the current partition's memory space based on the physical file name on the disk. Among them, the non-file data source is either MySQL or Elasticsearch; Spark writes the data from each partition to the memory space of the current partition at near-memory speed; The operating system periodically and asynchronously writes the data in the memory space of the current partition to physical files on the disk; Spark reads data using the memory space of the current partition plus the physical files on disk, and preheats the physical files on disk by loading as much data as possible from the physical files of each partition on disk into the leaf cache before converting the data into Spark DataFrames. Spark automatically adapts the optimal data join method based on the allocated executor memory and the size of the Spark DataFrame, including: Spark calculates the size of the storage memory based on the ratio of the sum of execution memory and storage memory to the total executor memory, and the ratio of storage memory to the sum of execution memory and storage memory. Calculate the proportion of Spark's DataFrame in the executor's storage memory based on the size of the storage memory and the size of Spark's DataFrame. If the proportion of Spark DataFrame's storage memory occupied by the executor is less than a specified threshold, then the Spark DataFrame is sent to each executor and stored in the memory of each executor. When performing data association, the data is read from the current executor. If the proportion of the Spark DataFrame occupying the executor's storage memory exceeds a specified threshold, then when performing data association, data will be read using the memory space of the current partition plus the physical files on the disk.

2. The method for improving the correlation performance of multi-source heterogeneous data according to claim 1, characterized in that, Add a file write success flag to the last line of data in the current partition's memory space.

3. The method for improving the correlation performance of multi-source heterogeneous data according to claim 1, characterized in that, After the data in the memory space of the current partition is written to the physical file on the disk, a status file is generated on the disk. Based on the data write time in the status file, it is determined whether Spark needs to fetch the non-file data source to each partition to generate files again.

4. An apparatus for improving the correlation performance of multi-source heterogeneous data, characterized in that, The device includes: The data writing module is used by Spark to perform concurrent partitioned readings from non-file data sources and establish a mapping between the addresses of physical files on disk and the virtual addresses of the current partition's memory space. This includes: reading the current partition number based on the partition field and the number of partitions, and reading data from the current partition; creating a new physical file on disk, with the physical filename containing the current partition number; initializing the virtual addresses of the current partition's memory space for the program process, and establishing a mapping between the addresses of physical files on disk and the virtual addresses of the current partition's memory space using the physical filename; where the non-file data source is MySQL or Elasticsearch; Spark writes the data from each partition to the current partition's memory space at near-memory speed; and the operating system periodically and asynchronously writes the data from the current partition's memory space to the physical file on disk. The data reading module is used by Spark to read data using the memory space of the current partition plus the physical files on the disk. It also preheats the physical files on the disk, loading as much data as possible from the physical files of each partition on the disk into the leaf cache, and then converts the data into Spark DataFrames. The data association adaptation module is used by Spark to automatically adapt the optimal data association method based on the allocated executor memory and the size of the Spark DataFrame. This includes: Spark calculating the storage memory size based on the ratio of the sum of execution memory and storage memory to the total executor memory, and the ratio of storage memory to the sum of execution memory and storage memory; calculating the ratio of the Spark DataFrame to the executor's storage memory based on the storage memory size and the Spark DataFrame size; if the ratio of the Spark DataFrame to the executor's storage memory is less than a specified threshold, the Spark DataFrame is sent to each executor and stored in the memory of each executor, and data is read from the current executor during data association; if the ratio of the Spark DataFrame to the executor's storage memory is greater than the specified threshold, data is read using the memory space of the current partition plus the physical files on the disk during data association.

5. The apparatus for improving the correlation performance of multi-source heterogeneous data according to claim 4, characterized in that, Add a file write success flag to the last line of data in the current partition's memory space.

6. The apparatus for improving the correlation performance of multi-source heterogeneous data according to claim 4, characterized in that, After the data in the memory space of the current partition is written to the physical file on the disk, a status file is generated on the disk. Based on the data write time in the status file, it is determined whether Spark needs to fetch the non-file data source to each partition to generate files again.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1-3.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that performs the method according to any one of claims 1-3.

Images