SCA source code analysis method and device based on user-defined mapping

Abstract

The invention discloses an SCA source code analysis method and device based on user-defined mapping, and the method comprises the steps: generating a self-increasing column corresponding to Hash column mapping, so as to build a mapping table; the fragment hash and the file hash in the Clickhome are read one by one, whether corresponding short hash exists in a mapping table or not is retrieved, if yes, the fragment hash and the file hash in the Clickhome are replaced with the corresponding short hash, and if not, the short hash is generated through corresponding mapping; analyzing and calculating the code snippets to obtain a fragment hash and a file hash; searching whether a corresponding short hash exists in the mapping table or not, if yes, taking out the short hash, and if not, correspondingly mapping to generate the short hash; and searching whether the same short hash exists in the Clickhome or not, if the same short hash exists in the Clickhome, carrying out merging, and if the same short hash exists in the Clickhome, storing the short hash in the Clickhome. According to the SCA source code analysis method based on the user-defined mapping, the strategy of mapping the long Hash into the short Hash is achieved through the user-defined mapping algorithm, the volume capacity of data can be effectively reduced, use of the storage space is optimized, and the overall cost of a system is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of SCA source code analysis, and in particular to a SCA source code analysis method and device based on user-defined mapping. Background Art

[0002] Due to the huge amount of data in the current SCA component code traceability, the hardware resources for storing related data are consumed greatly, and the cost is rising piece by piece. In addition, the use of common related compression algorithms on the market to compress the data cannot solve this problem. In addition, when parsing code snippets, the corresponding relationship is to calculate a fragment Hash for every 5 lines, corresponding to multiple file Hash, and although the generated Hash digits use 10 digits, the number of Hash digits is reduced compared to before, resulting in serious collisions between the generated Hash, the accuracy of the code parsing results is too low and the duplication is too high. A large amount of data redundancy has caused the code traceability data to occupy more than 30TB of disk capacity. Summary of the invention

[0003] The purpose of the present invention is to provide a SCA source code parsing method and device based on custom mapping, which uses a custom mapping algorithm to implement a strategy of mapping long hashes to short hashes, which can effectively reduce the volume capacity of data, optimize the use of storage space, and reduce the overall cost of the system.

[0004] In order to achieve the above object, the present invention discloses a SCA source code parsing method based on custom mapping, which comprises:

[0005] Generate an auto-increment column corresponding to Doris's Hash column mapping to establish a mapping table, wherein the short hash in the auto-increment column corresponds to the fragment hash and file hash in the Hash column;

[0006] Use the Flink engine to read the fragment hash and file hash in Clickhouse one by one, and search whether there is a corresponding short hash in the mapping table. If there is a corresponding short hash, replace the fragment hash and file hash in Clickhouse with the corresponding short hash. If there is no corresponding short hash, generate a short hash according to the corresponding mapping of the fragment hash and file hash, and add the mapping information to the mapping table;

[0007] Parse and calculate the code snippet according to the set number of lines to obtain the snippet hash and file hash;

[0008] According to the obtained fragment hash and file hash, the mapping table is searched to see whether there is a corresponding short hash. If there is a corresponding short hash, the corresponding short hash is taken out. If there is no corresponding short hash, a short hash is generated for the corresponding mapping, and the mapping information is added to the mapping table.

[0009] Check whether the same short hash exists in Clickhouse. If so, merge them. Otherwise, save the short hash to Clickhouse.

[0010] Furthermore, after the "corresponding to Doris's Hash column mapping to generate an auto-increment column to establish a mapping table", it also includes:

[0011] At least part of the mapping information in the mapping table is stored in a Caffice cache engine.

[0012] Furthermore, the “searching whether there is a corresponding short hash in the mapping table” includes:

[0013] Search whether there is a corresponding short hash in the mapping information of the Caffice cache engine. If there is a corresponding short hash, take out the corresponding short hash. If there is no corresponding short hash, search whether there is a corresponding short hash in the mapping table of Doris.

[0014] Furthermore, after “searching whether the same short hash exists in Clickhouse”, the following is also included:

[0015] Regularly check whether the mapping relationship between the fragment hash and file hash in the mapping table and the short hash in Clickhouse is consistent.

[0016] In order to achieve the above object, the present invention discloses an SCA source code parsing device based on custom mapping, which comprises:

[0017] The mapping generation module is used to generate an auto-increment column corresponding to Doris's Hash column mapping to establish a mapping table;

[0018] The first retrieval module is used to use the Flink engine to read the fragment hash and file hash in Clickhouse one by one, and search whether there is a corresponding short hash in the mapping table;

[0019] The parsing and calculation module is used to parse and calculate the code snippet according to the set number of lines to obtain the snippet hash and file hash;

[0020] A second retrieval module is used to retrieve whether there is a corresponding short hash in the mapping table according to the obtained fragment hash and file hash;

[0021] The third retrieval module is used to retrieve whether the same short hash exists in ClickHouse.

[0022] In order to achieve the above object, the present invention discloses an electronic device, which includes:

[0023] one or more processors;

[0024] One or more memories are used to store one or more programs. When one or more of the programs are executed by the processor, the processor implements the aforementioned SCA source code parsing method based on custom mapping.

[0025] In order to achieve the above object, the present invention discloses a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the SCA source code parsing method based on custom mapping as described above is implemented.

[0026] In the present application, a custom mapping algorithm is used to map long hashes to short hashes, and an auto-increment column is generated by mapping the corresponding Doris hash column to establish a mapping table so that the short hash corresponds to the fragment hash and the file hash. Then, the fragment hash and the file hash in Clickhouse are replaced with the short hash in the mapping table one by one, or the corresponding mapping generates a short hash, and the mapping information is updated to the mapping table. Then, radial parsing and calculation of the code fragment is started, and the mapping table is retrieved according to the obtained fragment hash and file hash. If the corresponding mapping information exists, the corresponding short hash is taken out, otherwise the corresponding mapping generates a short hash and updates the mapping table. Finally, Clickhouse is retrieved according to the short hash. If the same short hash exists, the data is merged, otherwise the short hash is saved. The above mapping and replacement of short hash operations can effectively reduce the volume capacity of data, optimize the use of storage space, and reduce the overall cost of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 The figure is a flow chart of an SCA source code parsing method based on custom mapping according to an embodiment of the present invention.

[0028] Figure 2 It is a module diagram of an SCA source code parsing device based on custom mapping according to an embodiment of the present invention.

[0029] Figure 3 4 is a system diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to explain the technical content, structural features, achieved objectives and effects of the present invention in detail, the following is a detailed description in conjunction with the implementation methods and the accompanying drawings.

[0031] Embodiment 1

[0032] See also Figure 1 The present invention discloses a SCA source code parsing method based on custom mapping, which includes:

[0033] 101. Generate an auto-increment column corresponding to Doris's Hash column mapping to establish a mapping table. The short hash in the auto-increment column corresponds to the fragment hash and file hash in the Hash column;

[0034] The corresponding mapping relationship is generated by utilizing the concurrent performance of Doris and its related index engine features, which is convenient for later data replacement.

[0035] It can be understood that by utilizing Doris's auto-increment sequence and Unique data model, an auto-increment column is generated with the Hash column as the corresponding mapping of the index column, so that the short Hash generated in the auto-increment column corresponds to the fragment Hash and file Hash in the Hash column, so as to construct a short Hash mapping table in Doris, but not limited to this.

[0036] Furthermore, after "Generating an auto-increment column corresponding to Doris's Hash column mapping to establish a mapping table", it also includes:

[0037] 1011. Store at least part of the mapping information in the mapping table in the Caffice cache engine.

[0038] Use local cache and other related strategies to implement dictionary mapping of later data to greatly improve the speed of code parsing.

[0039] It should be noted that the local Caffice cache engine is introduced into the SCA business to locally store the commonly used mapping information in the Hash mapping table. When retrieving data subsequently, the mapping information in the local Caffice cache engine is retrieved first to prevent all retrieval requests from going through the DB, thereby improving retrieval efficiency and performance.

[0040] 102. Use the Flink engine to read the fragment hash and file hash in Clickhouse one by one, and check whether there is a corresponding short hash in the mapping table. If there is a corresponding short hash, replace the fragment hash and file hash in Clickhouse with the corresponding short hash. If there is no corresponding short hash, generate a short hash based on the corresponding mapping of the fragment hash and file hash, and add the mapping information to the mapping table;

[0041] Shortening long hashes to short hashes can significantly reduce the requirements for storage and computing resources to achieve low-cost operation of the system.

[0042] Furthermore, “searching whether there is a corresponding short hash in the mapping table” includes:

[0043] 10. Check whether there is a corresponding short hash in the mapping information of the Caffice cache engine. If there is a corresponding short hash, take out the corresponding short hash. If there is no corresponding short hash, check whether there is a corresponding short hash in the mapping table of Doris.

[0044] It can be understood that the Flink batch processing engine is used to stream the stock data in Clickhouse, so as to batch update and replace the fragment hash as the key and the file hash as the value in the long hash with the corresponding short hash, and in the process of querying the mapping table, the Caffice cache engine will be searched first. If the relevant mapping information is not retrieved, the mapping table in Doris will be searched, and when the relevant mapping information is not stored in either, the mapping operation to generate a short hash is performed, and then the short hash result after the final replacement is saved in the new Clikchouse library, so as to facilitate the subsequent parsing to obtain new data and perform read and update operations, but it is not limited to this.

[0045] 103. Parse and calculate the code snippet according to the set number of lines to obtain the snippet hash and the file hash;

[0046] It can be understood that after the existing data is updated, new code snippet parsing begins to obtain incremental data. In this embodiment, the code snippet is parsed by calculating a snippet Hash as the Key for every 5 lines of code and using the corresponding file Hash as the value, but this is not limited to this.

[0047] 104. Search the mapping table to see if there is a corresponding short hash in the obtained fragment hash and file hash. If there is a corresponding short hash, take out the corresponding short hash. If there is no corresponding short hash, generate a short hash for the corresponding mapping, and add the mapping information to the mapping table.

[0048] Short hashes help reduce the complexity of data storage and retrieval, making queries faster.

[0049] Furthermore, “searching whether there is a corresponding short hash in the mapping table” includes:

[0050] 10. Check whether there is a corresponding short hash in the mapping information of the Caffice cache engine. If there is a corresponding short hash, take out the corresponding short hash. If there is no corresponding short hash, check whether there is a corresponding short hash in the mapping table of Doris.

[0051] It is understandable that the logic of the code snippet parsing device is modified so that after parsing the code snippet to obtain the snippet hash and file hash and storing them as Key and Value respectively, it first searches the local mapping relationship table for corresponding mapping information. If it exists, the corresponding short hash is taken out. If not, the mapping operation is performed to generate the short hash, and the mapping information is added to the Doris and Caffice cache engines. Then, the Key and Value are replaced with the short hash taken out or generated by mapping, and the new Clickhouse library is searched based on the replaced short hash to determine whether the same short hash exists therein. If it exists, it means that the short hash exists in multiple files, so the data is merged, but not limited to this.

[0052] 105. Check whether there is the same short hash in Clickhouse. If there is the same short hash, merge it, otherwise save the short hash to Clickhouse.

[0053] Efficient mapping can be used to avoid repeated storage of the same data, effectively reducing data redundancy. Combined with ClickHouse's high compression ratio, the overall capacity of SCA code traceability data can be effectively reduced.

[0054] Furthermore, after "checking whether the same short hash exists in ClickHouse", it also includes:

[0055] 106. Regularly check whether the mapping relationship between the fragment hash and file hash in the mapping table and the short hash in Clickhouse is consistent.

[0056] It is understandable that the mapping information of the mapping table is stored in both the Doris and Caffice cache engines. Therefore, the timed detection strategy set by the background daemon thread will run data batches every night to perform a hash comparison on the mapping table and the parsed code traceability file to verify whether the mapping relationship in Doris, Caffice, and Clickhouse is consistent, to ensure the ultimate consistency of the data results, but not limited to this.

[0057] In the present application, a custom mapping algorithm is used to map long hashes to short hashes, and an auto-increment column is generated by mapping the corresponding Doris hash column to establish a mapping table so that the short hash corresponds to the fragment hash and the file hash. Then, the fragment hash and the file hash in Clickhouse are replaced with the short hash in the mapping table one by one, or the corresponding mapping generates a short hash, and the mapping information is updated to the mapping table. Then, radial parsing and calculation of the code fragment is started, and the mapping table is retrieved according to the obtained fragment hash and file hash. If the corresponding mapping information exists, the corresponding short hash is taken out, otherwise the corresponding mapping generates a short hash and updates the mapping table. Finally, Clickhouse is retrieved according to the short hash. If the same short hash exists, the data is merged, otherwise the short hash is saved. The above mapping and replacement of short hash operations can effectively reduce the volume capacity of data, optimize the use of storage space, and reduce the overall cost of the system.

[0058] Embodiment 2

[0059] See also Figure 1 and Figure 2 The present invention discloses an SCA source code parsing device based on custom mapping, which includes:

[0060] A mapping generation module 201 is used to generate an auto-increment column corresponding to Doris's Hash column mapping to establish a mapping table;

[0061] The first retrieval module 202 is used to use the Flink engine to read the fragment hash and file hash in Clickhouse one by one, and search whether there is a corresponding short hash in the mapping table;

[0062] The parsing and calculating module 203 is used to parse and calculate the code snippet according to the set number of lines to obtain the snippet hash and the file hash;

[0063] A second retrieval module 204 is used to retrieve whether there is a corresponding short hash in the mapping table according to the obtained fragment hash and file hash;

[0064] The third retrieval module 205 is used to retrieve whether the same short hash exists in Clickhouse.

[0065] Embodiment 3

[0066] See also Figure 1 and Figure 3 , the present invention discloses an electronic device, comprising:

[0067] One or more processors 301;

[0068] The one or more memories 302 are used to store one or more programs. When the one or more programs are executed by the processor, the processor implements the aforementioned SCA source code parsing method based on custom mapping.

[0069] Embodiment 4

[0070] The embodiment of the present application discloses a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the SCA source code parsing method based on custom mapping as described above is implemented.

[0071] Embodiment 5

[0072] The embodiment of the present application discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of an electronic device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the electronic device executes the above-mentioned SCA source code parsing method based on custom mapping.

[0073] It should be understood that in the embodiments of the present application, the processor referred to may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0074] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by hardware related to computer program instructions, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0075] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of rights of the present invention. Therefore, equivalent changes made according to the scope of the patent application of the present invention are still within the scope covered by the present invention.

Claims

1. A SCA source code parsing method based on custom mapping, characterized in that: include: Generate an auto-increment column corresponding to Doris's Hash column mapping to establish a mapping table, wherein the short hash in the auto-increment column corresponds to the fragment hash and file hash in the Hash column; Use the Flink engine to read the fragment hash and file hash in Clickhouse one by one, and search whether there is a corresponding short hash in the mapping table. If there is a corresponding short hash, replace the fragment hash and file hash in Clickhouse with the corresponding short hash. If there is no corresponding short hash, generate a short hash according to the corresponding mapping of the fragment hash and file hash, and add the mapping information to the mapping table; Parse and calculate the code snippet according to the set number of lines to obtain the snippet hash and file hash; According to the obtained fragment hash and file hash, the mapping table is searched to see whether there is a corresponding short hash. If there is a corresponding short hash, the corresponding short hash is taken out. If there is no corresponding short hash, a short hash is generated for the corresponding mapping, and the mapping information is added to the mapping table. Check whether the same short hash exists in Clickhouse. If so, merge them. Otherwise, save the short hash to Clickhouse.

2. The SCA source code parsing method based on custom mapping according to claim 1 is characterized in that: After the "Hash column mapping corresponding to Doris generates an auto-increment column to establish a mapping table", it also includes: At least part of the mapping information in the mapping table is stored in a Caffice cache engine.

3. The SCA source code parsing method based on custom mapping according to claim 2 is characterized in that: The "searching whether there is a corresponding short hash in the mapping table" includes: Search whether there is a corresponding short hash in the mapping information of the Caffice cache engine. If there is a corresponding short hash, take out the corresponding short hash. If there is no corresponding short hash, search whether there is a corresponding short hash in the mapping table of Doris.

4. The SCA source code parsing method based on custom mapping according to claim 1 is characterized in that: After "Searching whether the same short hash exists in ClickHouse", it also includes: Regularly check whether the mapping relationship between the fragment hash and file hash in the mapping table and the short hash in Clickhouse is consistent.

5. An SCA source code parsing device based on custom mapping, characterized in that: include: The mapping generation module is used to generate an auto-increment column corresponding to Doris's Hash column mapping to establish a mapping table; The first retrieval module is used to use the Flink engine to read the fragment hash and file hash in Clickhouse one by one, and search whether there is a corresponding short hash in the mapping table; The parsing and calculation module is used to parse and calculate the code snippet according to the set number of lines to obtain the snippet hash and file hash; A second retrieval module is used to retrieve whether there is a corresponding short hash in the mapping table according to the obtained fragment hash and file hash; The third retrieval module is used to retrieve whether the same short hash exists in ClickHouse.

6. An electronic device, characterized in that: include: one or more processors; One or more memories are used to store one or more programs. When one or more of the programs are executed by the processor, the processor implements the SCA source code parsing method based on custom mapping as described in any one of claims 1 to 4.

7. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the SCA source code parsing method based on custom mapping as described in any one of claims 1 to 4 is implemented.

Images