SCA source code traceability updating method and device based on multiple storage engines

Abstract

The invention discloses an SCA source code traceability updating method and device based on multiple storage engines. The updating method comprises the following steps: acquiring a core version file of a to-be-analyzed component at regular time; analyzing the code of the core version file to obtain a plurality of file hash; de-duplicating the plurality of file hash by utilizing a set set of redis, and storing the de-duplicated file hash in the redis in a key-value form; the number of key values in the redis is detected regularly, whether the number of the key values is larger than a set threshold value or not is judged, and if the number of the key values is larger than the set threshold value, file hash in the redis is migrated to a to-be-merged table of the doris; combining the file hash in the to-be-combined table in batches by using the flink to obtain a combined set; repeating the operations of analysis, deduplication and merging until the analysis of the to-be-analyzed component is completed; and migrating the file hash of the combined set to the clikchouse through the flink. According to the SCA source code traceability updating method based on the multiple storage engines, the whole updating operation is split, and the overall updating efficiency of the SCA source code traceability data is improved in the mode that the multiple storage engines are executed in batches.

Description

Technical Field

[0001] The present invention relates to the technical field of SCA source code tracing and updating, and in particular to a SCA source code tracing and updating method and device based on multiple storage engines. Background Art

[0002] At present, in the process of SCA source code traceability update, the total amount of source code traceability to be parsed by SCA is too large, and the data volume of a single table can reach tens of billions, so it is difficult to complete the update of related data through ordinary methods. Due to the characteristics of the clickhouse component itself, its support for update and deletion operations is general, and it does not support related modification transactions. It is difficult to directly operate the clickhouse component to complete the SCA source code traceability update. Therefore, the update of SCA source code traceability has always had the efficiency problem of slow big data update. Summary of the invention

[0003] The purpose of the present invention is to provide an SCA source code traceability update method and device based on multiple storage engines, which improves the overall update efficiency of SCA source code traceability data by splitting the entire update operation and executing it in batches using multiple storage engines.

[0004] In order to achieve the above object, the present invention discloses an SCA source code traceability and update method based on multiple storage engines, which includes:

[0005] Regularly obtain the core version file of the component to be parsed;

[0006] Parsing the code of the core version file to obtain multiple file hashes;

[0007] De-duplicate the hashes of the multiple files using the set collection of redis, and store them in the form of key-value in redis;

[0008] Regularly detect the number of key values ​​in redis and determine whether the number of key values ​​is greater than the set threshold. If it is greater than the set threshold, migrate the file hash in redis to the table to be merged in doris;

[0009] Use Flink to merge the hashes of the files in the table to be merged in batches by searching the table to be merged and Climax to obtain a merged set;

[0010] Repeat the above parsing, deduplication, and merging operations until the parsing of the component to be parsed is completed;

[0011] Use Flink to migrate the file hash of the merged set to Clikchouse.

[0012] Furthermore, the “regularly obtaining the core version file of the component to be parsed” includes:

[0013] Regularly read the collected information of the components to be parsed in MongoDB;

[0014] Obtaining a version file of the component to be parsed according to the collected information;

[0015] The obtained version files are filtered according to the filtering rules to retain the core version files.

[0016] Furthermore, the "migrating the file hash in redis to the table to be merged in doris" includes:

[0017] Send parsing pause information through redis stream;

[0018] Use steam-load to migrate the file hash in redis to the table to be merged in doris;

[0019] Clear the data in redis.

[0020] Furthermore, before the “using Flink to retrieve the table to be merged and CLICKCHOUNT to merge the file hashes in the table to be merged in batches to obtain a merged set”, it also includes:

[0021] Flink is used to periodically detect whether the table to be merged has data to be merged. If the data to be merged exists, Flink is used to merge the file hashes in the table to be merged in batches.

[0022] Furthermore, after the "obtaining the merged set", the method further includes:

[0023] Send parsing continuation information through redis stream.

[0024] In order to achieve the above object, the present invention discloses an SCA source code traceability and update device based on multiple storage engines, which is characterized by comprising:

[0025] The acquisition module is used to periodically obtain the core version file of the component to be parsed;

[0026] A parsing module, used for parsing the code of the core version file to obtain multiple file hashes;

[0027] A deduplication module is used to deduplicate the hashes of the plurality of files by using a set collection of redis, and store them in the form of key-value in redis;

[0028] The first migration module is used to periodically detect the number of key values ​​in redis and determine whether the number of key values ​​is greater than a set threshold. If it is greater than the set threshold, the file hash in redis is migrated to the table to be merged in doris;

[0029] A merging module is used to merge the hashes of the files in the table to be merged in batches by using Flink to retrieve the table to be merged and Clikchouse to obtain a merged set;

[0030] The second migration module is used to migrate the file hash of the merged set to clinkchouse through flink.

[0031] In order to achieve the above object, the present invention discloses an electronic device, which includes:

[0032] one or more processors;

[0033] One or more memories are used to store one or more programs. When one or more of the programs are executed by the processor, the processor implements the SCA source code traceability and update method based on multiple storage engines as described above.

[0034] In order to achieve the above-mentioned object, the present invention discloses a computer-readable storage medium having a program stored thereon, and when the program is executed by a processor, the SCA source code traceability and update method based on multiple storage engines as described above is implemented.

[0035] In this application, the overall update operation of SCA source code traceability is performed in batches through multiple storage engines. First, the core version file of the component to be parsed is obtained regularly to parse its code to obtain multiple file hashes, and the file hash is deduplicated using the set collection of redis. Then, the number of key values ​​in redis is detected regularly to determine whether to migrate the file hash in redis to the table to be merged in doris, so as to use the batch processing capability of flink to merge the file hashes in the table to be merged and obtain a merged set. The above process is repeated until the parsing of the component to be parsed is completed. Finally, the file hash of the merged set is migrated to clikchouse through flink, that is, the update of the SCA source code traceability data is completed. The entire update operation is split, and the update operation is performed in batches using multiple storage engines, which can effectively improve the overall update efficiency of the SCA source code traceability data and solve the problem of clikchouse update mutation. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 The present invention is a flowchart of an SCA source code traceability and update method based on multiple storage engines according to an embodiment of the present invention.

[0037] Figure 2 This is a module diagram of an SCA source code traceability and update device based on multiple storage engines according to an embodiment of the present invention.

[0038] Figure 3 4 is a system diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to explain the technical content, structural features, achieved objectives and effects of the present invention in detail, the following is a detailed description in conjunction with the implementation methods and the accompanying drawings.

[0040] Embodiment 1

[0041] See also Figure 1 The present invention discloses a SCA source code traceability and update method based on multiple storage engines, which includes:

[0042] 101. Regularly obtain the core version file of the component to be parsed;

[0043] Furthermore, “regularly obtaining the core version file of the component to be parsed” includes:

[0044] 1011. Regularly read the collected information of the components to be parsed in MongoDB;

[0045] 1012. Obtain a version file of the component to be parsed according to the collected information;

[0046] 1013. Filter the obtained version files according to the screening rules to retain the core version files.

[0047] It should be noted that the program uses the XXL-JOB task component to periodically read the collection information of the components to be parsed in the MongoDB database. The collection information includes the version information of the components that have been downloaded. The corresponding component version files are decompressed based on the collection information, and the version files are filtered according to the common core version file information to retain the corresponding core version files in the component version files, but not limited to this.

[0048] 102. Parse the code of the core version file to obtain multiple file hashes;

[0049] It can be understood that the basic logic for parsing the code snippet information of a version file is to parse a hash value corresponding to a set number of lines of code as the file hash of the version file. For example, a snippet hash is calculated for every 5 lines of code as the file hash of the version file, or, for a version file with 1000 lines of code, 1000 line hashes are calculated as the file hash of the version file. Moreover, the difference between different version files of the same component is usually not large, so for the same snippet hash or line hash, a file hash merge operation needs to be performed, but not limited to this.

[0050] 103. Use the set collection of redis to deduplicate multiple file hashes and store them in redis in the form of key-value;

[0051] It can be understood that by directly inserting the file hash of the number of code lines * the number of versions into redis, the Set collection of the redis storage layer can quickly and automatically perform deduplication operations on the file hash, achieving real-time deduplication while effectively accelerating the deduplication efficiency, avoiding the conventional update practice: the need to query and retrieve the database and perform deduplication operations the same number of times as the number of code lines * the number of versions, but not limited to this.

[0052] 104. Regularly detect the number of key values ​​in redis and determine whether the number of key values ​​is greater than the set threshold. If it is greater than the set threshold, migrate the file hash in redis to the table to be merged in doris;

[0053] Furthermore, "migrating the file hash in redis to the merged table of doris" includes:

[0054] 1041. Send parsing pause information through redis stream;

[0055] 1042. Use steam-load to migrate the file hash in redis to the table to be merged in doris;

[0056] 1043. Clear the data in redis.

[0057] It can be understood that the program is provided with a parsing thread to perform the operation of parsing file hashes, and a daemon thread is provided in the background of the program. The daemon thread periodically detects the number of key values ​​currently related to the file hash in the redis storage layer. For example, the threshold value can be set to five million. When the number of key values ​​exceeds five million, the daemon thread sends a parsing pause message to the parsing thread through the redis stream to inform it to suspend the parsing operation. At the same time, the file hash data in redis is quickly migrated to the table to be merged (temporary table) of doris through the stream-load of redis to facilitate subsequent data merging work, and after the data migration is completed, the data in the table to be merged of redis is cleared to release the memory of redis and wait for the next batch of data to be merged to be written, but not limited to this.

[0058] 105. Use Flink to merge the file hashes in the table to be merged in batches by retrieving the table to be merged and clikchouse to obtain a merged set;

[0059] Furthermore, before "using Flink to retrieve the table to be merged and CLICKCHOO to merge the file hashes in the table to be merged in batches to obtain a merged set", it also includes:

[0060] 1050. Use Flink to periodically detect whether there is data to be merged in the table to be merged. If there is data to be merged, use Flink to merge the file hashes in the table to be merged in batches.

[0061] It can be understood that flink periodically detects whether there are file hashes to be merged in doris's table to be merged. If it is detected that there are data to be merged, flink will read these data to be merged in batches. For example, the file hashes in the table to be merged are read in batches of 5,000 data, and then clikchouse is searched based on the read data to determine which file hashes read and in clikchouse are the same, and then these identical file hashes in the table to be merged are de-duplicated and merged. The batch streaming capability of flink is used to process the data in the table to be merged in batches, which can make full use of the computing power of the CPU, quickly merge data, and improve data processing efficiency, but it is not limited to this.

[0062] Furthermore, after "obtaining the merged set", it also includes:

[0063] 1051. Send parsing continuation information through redis stream.

[0064] 106. Repeat the above parsing, deduplication, and merging operations until the parsing of the component to be parsed is completed;

[0065] It can be understood that when Flink completes the operation of merging the file hash data in Doris, it will send parsing continuation information to the parsing thread through the Redis stream to perform the above parsing, deduplication and merging operations again, and obtain all updated data of the components to be parsed.

[0066] 107. Use Flink to migrate the merged set file hash to Clirikchouse.

[0067] It can be understood that after the parsing of all components to be parsed is completed through the above update operation, all updated data in the merged set is migrated to ClickHouse through Flink, thus completing the overall update operation of the SCA source code traceability data.

[0068] In this application, the overall update operation of SCA source code traceability is performed in batches through multiple storage engines. First, the core version file of the component to be parsed is obtained regularly to parse its code to obtain multiple file hashes, and the file hash is deduplicated using the set collection of redis. Then, the number of key values ​​in redis is detected regularly to determine whether to migrate the file hash in redis to the table to be merged of doris, so as to use the batch processing capability of flink to merge the file hash in the table to be merged and obtain a merged set. The above process is repeated until the parsing of the component to be parsed is completed. Finally, the file hash of the merged set is migrated to clikchouse through flink, that is, the update of SCA source code traceability data is completed. The advantages of multiple storage engines are utilized, such as the Set deduplication mechanism of redis, the data aggregation capability of doris, and the stream batch processing mechanism of flink, the entire update operation is split, and the original separate process of updating clickhouse data is converted into a multi-storage layer batch execution of SCA code traceability update operation, which can effectively improve the overall update efficiency of SCA source code traceability data, and solve the problem of update deletion mutation related to clikchouse.

[0069] Embodiment 2

[0070] See also Figure 1 and Figure 2 The present invention discloses an SCA source code traceability and update device based on multiple storage engines, which includes:

[0071] The acquisition module 201 is used to periodically acquire the core version file of the component to be parsed;

[0072] The parsing module 202 is used to parse the code of the core version file to obtain multiple file hashes;

[0073] The deduplication module 203 is used to deduplicate multiple file hashes by using the set collection of redis, and store them in the form of key-value in redis;

[0074] The first migration module 204 is used to periodically detect the number of key values ​​in redis and determine whether the number of key values ​​is greater than a set threshold. If it is greater than the set threshold, the file hash in redis is migrated to the to-be-merged table of doris;

[0075] The merging module 205 is used to merge the hashes of the files in the table to be merged in batches by using Flink to retrieve the table to be merged and Climax to obtain a merged set;

[0076] The second migration module 206 is used to migrate the file hash of the merged set to the clinkchouse through Flink.

[0077] Embodiment 3

[0078] See also Figure 1 and Figure 3 , the present invention discloses an electronic device, comprising:

[0079] One or more processors 301;

[0080] One or more memories 302 are used to store one or more programs. When the one or more programs are executed by the processor, the processor implements the aforementioned SCA source code traceability and update method based on multiple storage engines.

[0081] Embodiment 4

[0082] The embodiment of the present application discloses a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the SCA source code traceability and update method based on multiple storage engines as described above is implemented.

[0083] Embodiment 5

[0084] The embodiment of the present application discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of an electronic device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the electronic device executes the above-mentioned SCA source code traceability update method based on multiple storage engines.

[0085] It should be understood that in the embodiments of the present application, the processor referred to may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0086] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by hardware related to computer program instructions, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0087] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of rights of the present invention. Therefore, equivalent changes made according to the scope of the patent application of the present invention are still within the scope covered by the present invention.

Claims

1. A SCA source code traceability and update method based on multiple storage engines, characterized in that: include: Regularly obtain the core version file of the component to be parsed; Parsing the code of the core version file to obtain multiple file hashes; De-duplicate the hashes of the multiple files using the set collection of redis, and store them in the form of key-value in redis; Regularly detect the number of key values ​​in redis and determine whether the number of key values ​​is greater than the set threshold. If it is greater than the set threshold, migrate the file hash in redis to the table to be merged in doris; Use Flink to merge the hashes of the files in the table to be merged in batches by searching the table to be merged and Climax to obtain a merged set; Repeat the above parsing, deduplication, and merging operations until the parsing of the component to be parsed is completed; Use Flink to migrate the file hash of the merged set to Clikchouse.

2. The SCA source code tracing and updating method based on multiple storage engines according to claim 1 is characterized in that: The "regularly obtaining the core version file of the component to be parsed" includes: Regularly read the collected information of the components to be parsed in MongoDB; Obtaining a version file of the component to be parsed according to the collected information; The obtained version files are filtered according to the filtering rules to retain the core version files.

3. The SCA source code tracing and updating method based on multiple storage engines according to claim 1 is characterized in that: The "migrating the file hash in redis to the table to be merged in doris" includes: Send parsing pause information through redis stream; Use steam-load to migrate the file hash in redis to the table to be merged in doris; Clear the data in redis.

4. The SCA source code tracing and updating method based on multiple storage engines according to claim 1 is characterized in that: Before the "using Flink to retrieve the table to be merged and CLICKCHOO to merge the file hashes in the table to be merged in batches to obtain a merged set", it also includes: Flink is used to periodically detect whether the table to be merged has data to be merged. If the data to be merged exists, Flink is used to merge the file hashes in the table to be merged in batches.

5. The SCA source code tracing and updating method based on multiple storage engines according to claim 1 is characterized in that: After the "obtaining the merged set", it also includes: Send parsing continuation information through redis stream.

6. An SCA source code traceability and update device based on multiple storage engines, characterized in that: include: The acquisition module is used to periodically obtain the core version file of the component to be parsed; A parsing module, used for parsing the code of the core version file to obtain multiple file hashes; A deduplication module is used to deduplicate the hashes of the plurality of files by using a set collection of redis, and store them in the form of key-value in redis; The first migration module is used to periodically detect the number of key values ​​in redis and determine whether the number of key values ​​is greater than a set threshold. If it is greater than the set threshold, the file hash in redis is migrated to the table to be merged in doris; A merging module is used to merge the hashes of the files in the table to be merged in batches by using Flink to retrieve the table to be merged and Clikchouse to obtain a merged set; The second migration module is used to migrate the file hash of the merged set to clinkchouse through flink.

7. An electronic device, characterized in that: include: one or more processors; One or more memories are used to store one or more programs. When one or more of the programs are executed by the processor, the processor implements the SCA source code traceability and update method based on multiple storage engines as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the SCA source code traceability and update method based on multiple storage engines as described in any one of claims 1 to 5 is implemented.

Images