SCA source code analysis method and device based on scheduler, equipment and storage medium
In the process of SCA code snippet parsing, the scheduler integrates acquisition, calculation, migration, merge and exception nodes, and solves the inefficiency problem in the existing technology, and realizes automated process management and efficient SCA code snippet parsing.
Patent Information
- Application Number
- CN202411232785.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2025-02-07
AI Technical Summary
During the parsing of existing SCA code snippets, the operation steps are different, resulting in a reduction in overall efficiency and the inability to process and continue to advance in a timely manner.
Through scheduler node flow, acquisition, calculation, migration, merging and exception nodes are integrated to realize automated process management and task associations, and improve the efficiency of SCA code snippet parsing.
It effectively improves the overall efficiency of SCA code snippet parsing, realizes automated process management, and avoids delays and redundancy between operation steps.
Smart Images
Figure CN119397526A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of SCA source code analysis, and in particular to a scheduler-based SCA source code analysis method, device, equipment and storage medium. Background Art
[0002] In the existing SCA code snippet parsing process, the basic operation processes include collection, parsing, merging, migration, deduplication, exception handling, etc., and the corresponding operation steps in different processing stages are likely to be developed in different languages. Therefore, it is necessary to monitor the results of the previous operation by setting up timed task detection and other measures to trigger the start of the next process. Such a setting is prone to a certain extent that operations cannot be processed in time and the process is difficult to continue to advance, resulting in a decrease in the overall efficiency of the SCA code snippet parsing function. Summary of the invention
[0003] The purpose of the present invention is to provide a scheduler-based SCA source code parsing method, device, equipment and storage medium, which integrates different operations through the flow of scheduler nodes, realizes automated process management and the correlation between each independent task, and effectively improves the overall efficiency of SCA code fragment parsing.
[0004] In order to achieve the above object, the present invention discloses a SCA source code parsing method based on a scheduler, which uses the scheduler to establish collection nodes, calculation nodes, migration nodes, merging nodes and abnormal nodes. The parsing method includes:
[0005] Executing the collection node to collect a batch of component source code files of the components to be parsed, and starting the calculation node;
[0006] Execute the computing node, parse the file hash of the component source code file, write it to Redis, and then start the migration node;
[0007] Execute the migration node, migrate the file hash in Redis to Doris, and start the merge node;
[0008] Execute the merge node to merge and remove duplicate file hashes in Doris, and return to start the computing node until the parsing of the component source code files of the current batch is completed, and then return to start the acquisition node to start processing a new batch of components to be parsed;
[0009] When executing each node, the abnormal node is executed synchronously, and the execution of each node is monitored. If an abnormality occurs in the monitored node, a node retry mechanism is executed.
[0010] Furthermore, the “collecting a batch of component source code files of components to be parsed” includes:
[0011] According to the set number of components, multiple components to be parsed are grouped into a batch of components to be parsed;
[0012] Download the component source code files of the batch of components to be parsed from the corresponding data source.
[0013] Furthermore, the “parsing the file hash of the component source code file, writing it into Redis, and then starting the migration node” includes:
[0014] Calculate a fragment hash according to the set number of lines as the file hash of the component source code file;
[0015] Write the calculated file hash to Redis and deduplicate the same file hash in Redis;
[0016] It is determined whether the memory of Redis reaches a set threshold. If so, the calculation of the fragment hash is suspended and the migration node is started. If not, the calculation of the fragment hash continues.
[0017] Furthermore, the “migrating the file hash in Redis to Doris” includes:
[0018] Use Flink to synchronize the file hash in Redis to the to-be-merged column in Doris;
[0019] Clear the file hash in Redis.
[0020] Furthermore, the “merging and deduplicating the file hashes in Doris” includes:
[0021] Use Doris's UDF function to merge the result column and the file hash in the column to be merged, and then deduplicate the data;
[0022] The deduplicated file hash is saved to the result column, and the file hash in the to-be-merged column is cleared.
[0023] Furthermore, the "node retry mechanism" includes:
[0024] Re-execute the current node three times. If it fails three times, record the current execution result and send a notification message.
[0025] The parsing of the component source code files of the current batch is terminated, and the collection node is returned to start the parsing of the component source code files of a new batch.
[0026] In order to achieve the above object, the present invention discloses a SCA source code parsing device based on a scheduler, which is equipped with a collection node, a calculation node, a migration node, a merging node and an abnormal node. The parsing device includes:
[0027] A collection module, used to execute the collection node, collect a batch of component source code files of the components to be parsed, and start the calculation node;
[0028] A computing module, used to execute the computing node, parse the file hash of the component source code file, write it to Redis, and then start the migration node;
[0029] Migration module, used to execute the migration node, migrate the file hash in Redis to Doris, and start the merge node;
[0030] A merging module is used to execute the merging node, merge and de-duplicate the file hashes in Doris, and return to start the computing node or the collection node;
[0031] The exception module is used to synchronously execute the abnormal node when executing each node, monitor the execution of each node, and execute the node retry mechanism if an abnormality occurs in the monitored node.
[0032] In order to achieve the above object, the present invention discloses an electronic device, which includes:
[0033] one or more processors;
[0034] One or more memories are used to store one or more programs. When one or more of the programs are executed by the processor, the processor implements the aforementioned scheduler-based SCA source code parsing method.
[0035] In order to achieve the above object, the present invention discloses a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned scheduler-based SCA source code parsing method is implemented.
[0036] In this application, a scheduler is used to establish a collection node, a computing node, a migration node, a merging node, and an abnormal node to correlate each independent task of SCA source code parsing, execute the collection node, collect a batch of component source code files of the components to be parsed, and start the computing node; execute the computing node, parse the file hash of the component source code file, and write it to Redis, and then start the migration node; execute the migration node, migrate the file hash in Redis to Doris, and start the merging node; execute the merging node, merge and remove the file hash in Doris, and return to start the computing node until the parsing of the component source code files of the current batch is completed, and then return to start the collection node to start processing a new batch of components to be parsed; and execute the abnormal node synchronously in each node, monitor the execution of each node, and execute the node retry mechanism when the monitored node is abnormal. Different operations are integrated by the above-mentioned scheduler node flow method to achieve automated process management and effectively improve the overall efficiency of SCA code snippet parsing. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 The figure is a flow chart of the SCA source code parsing method based on the scheduler according to an embodiment of the present invention.
[0038] Figure 2 It is a module diagram of the SCA source code parsing device based on the scheduler according to an embodiment of the present invention.
[0039] Figure 3 4 is a system diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0040] In order to explain the technical content, structural features, achieved objectives and effects of the present invention in detail, the following is a detailed description in conjunction with the implementation methods and the accompanying drawings.
[0041] Embodiment 1
[0042] See also Figure 1 The present invention discloses a SCA source code parsing method based on a scheduler, which uses the scheduler to establish collection nodes, calculation nodes, migration nodes, merging nodes and abnormal nodes.
[0043] As a data processing platform, the scheduler is versatile and easily expandable, and can adapt to the serial scheduling operations of different processes by changing the nodes.
[0044] It should be noted that in this embodiment, the Dolphin Scheduler is used, but it is not limited to this. Multiple task nodes are established in combination with SCA code traceability related services, and a task node of the Dolphin Scheduler is completed and the next node is automatically started, so that the upper and lower nodes can respond quickly.
[0045] Parsing methods include:
[0046] 101. Execute the collection node to collect a batch of component source code files of the components to be parsed, and start the computing node;
[0047] Furthermore, “collecting a batch of component source code files of components to be parsed” includes:
[0048] 1011. Grouping multiple components to be parsed into a batch of components to be parsed according to a set number of components;
[0049] 1012. Download the component source code files of the batch of components to be parsed from the corresponding data source.
[0050] It can be understood that in the Dolphin Scheduler, a custom Python script is created to define the collection node as the starting node. In this embodiment, the number of components is set to one hundred, that is, one hundred components to be parsed are taken as a batch, and the source code file information of the components to be parsed in this batch is collected from different data sources, and the component source code files are downloaded, but not limited to this.
[0051] 102. Execute the computing node, parse the file hash of the component source code file, write it to Redis, and then start the migration node;
[0052] Furthermore, “parsing the file hash of the component source code file, writing it to Redis, and then starting the migration node” includes:
[0053] 1021. Calculate a fragment hash according to the set number of lines as the file hash of the component source code file;
[0054] It can be understood that when a batch of component source code files is downloaded, the scheduler executes the computing node to call the Java parsing device to perform parsing and perform code fragment data analysis on the downloaded component source code files. In this embodiment, the number of lines is set to five, that is, a fragment Hash is calculated for every five lines of code, which corresponds to the file Hash of the component source code file, but is not limited to this.
[0055] 1022. Write the calculated file hash into Redis, and deduplicate the same file hash in Redis;
[0056] It is understandable that the set feature of Redis is used to deduplicate data on the same file hash.
[0057] 1023. Determine whether the memory of Redis reaches a set threshold. If so, suspend the calculation of the fragment hash and start the migration node. If not, continue to calculate the fragment hash.
[0058] It is understandable that a daemon thread is set in the background to synchronously monitor the memory usage of Redis while writing the file hash to Redis. In this embodiment, the threshold is set to 20GB, that is, when the memory of Redis reaches 20GB, operation 1021 will be suspended and the migration node of the Dolphin Scheduler will be started to execute operation 103, but this is not limited to it.
[0059] 103. Execute the migration node, migrate the file hash in Redis to Doris, and start the merge node;
[0060] Furthermore, “Migrate file hashes from Redis to Doris” includes:
[0061] 1031. Use Flink to synchronize the file hash in Redis to the to-be-merged column of Doris;
[0062] 1032. Clear the file hash in Redis.
[0063] It is understandable that the Dolphin Scheduler creates a Flink script to define the data migration node, and migrates 20GB of parsed data in Redis to Doris through the stream reading method of Flink, a big data component that integrates stream and batch, but it is not limited to this.
[0064] 104. Execute the merge node to merge and remove duplicate file hashes in Doris, and return to the start calculation node until the parsing of the component source code files of the current batch is completed, and then return to the start collection node to start processing a new batch of components to be parsed;
[0065] By utilizing the characteristics of the Dolphin Scheduler and the definition of nodes, the operations at each stage of SCA source code parsing are integrated and associated to form automated closed-loop management, and big data components are integrated, which can effectively simplify the complexity of the SCA business process and thereby improve the overall parsing performance and efficiency of SCA.
[0066] Furthermore, "merging and deduplicating the file hashes in Doris" includes:
[0067] 1041. Use Doris's UDF function to merge the result column and the file hash in the column to be merged, and then deduplicate the data;
[0068] 1042. Save the deduplicated file hash to the result column, and clear the file hash in the to-be-merged column.
[0069] It can be understood that a SQL script is created in the Dolphin Scheduler to define a data merge node. After the data merge is completed, the operation 102 is returned to start the computing node, and then the computing node, migration node and merge node are repeatedly executed in sequence to complete the calculation of the code snippet data of all component source code files in the current batch, effectively realizing the automatic switching of each stage in the SCA code snippet parsing process and avoiding redundant operations in the process, but not limited to this.
[0070] 105. When executing each node, execute the abnormal node synchronously and monitor the execution of each node. If the monitored node is abnormal, execute the node retry mechanism.
[0071] Furthermore, the "node retry mechanism" includes:
[0072] 1051. Re-execute the current node three times. If all three times fail, record the current execution result and send a notification message.
[0073] 1052. End the parsing of the component source code files of the current batch, and return to the start collection node to start parsing of a new batch of component source code files.
[0074] It is understandable that the Dolphin Scheduler monitors the SCA source code parsing for exceptions through abnormal nodes. When an exception occurs, the operation will be retried. If the node can be executed normally again within three retries, no other processing will be performed. If the three retries fail, the administrator will be notified and the parsing of the next batch of component source code files will be started directly, but this is not limited to this.
[0075] In this application, a scheduler is used to establish a collection node, a computing node, a migration node, a merging node, and an abnormal node to correlate each independent task of SCA source code parsing, execute the collection node, collect a batch of component source code files of the components to be parsed, and start the computing node; execute the computing node, parse the file hash of the component source code file, and write it to Redis, and then start the migration node; execute the migration node, migrate the file hash in Redis to Doris, and start the merging node; execute the merging node, merge and remove the file hash in Doris, and return to start the computing node until the parsing of the component source code files of the current batch is completed, and then return to start the collection node to start processing a new batch of components to be parsed; and execute the abnormal node synchronously in each node, monitor the execution of each node, and execute the node retry mechanism when the monitored node is abnormal. Different operations are integrated by the above-mentioned scheduler node flow method to achieve automated process management and effectively improve the overall efficiency of SCA code snippet parsing.
[0076] Embodiment 2
[0077] See also Figure 1 and Figure 2 The present invention discloses a SCA source code parsing device based on a scheduler, which is equipped with a collection node, a calculation node, a migration node, a merging node and an abnormal node. The parsing device includes:
[0078] The collection module 201 is used to execute the collection node, collect a batch of component source code files of the components to be analyzed, and start the computing node;
[0079] The computing module 202 is used to execute the computing node, parse the file hash of the component source code file, write it to Redis, and then start the migration node;
[0080] Migration module 203, used to execute the migration node, migrate the file hash in Redis to Doris, and start the merge node;
[0081] The merging module 204 is used to execute the merging node, merge and remove duplicates of the file hashes in Doris, and return to start the computing node or the collection node;
[0082] The exception module 205 is used to synchronously execute the abnormal node when executing each node, monitor the execution of each node, and execute the node retry mechanism if an abnormality occurs in the monitored node.
[0083] Embodiment 3
[0084] See also Figure 1 and Figure 3 , the present invention discloses an electronic device, comprising:
[0085] One or more processors 301;
[0086] One or more memories 302 are used to store one or more programs. When the one or more programs are executed by the processor, the processor implements the aforementioned scheduler-based SCA source code parsing method.
[0087] Embodiment 4
[0088] The embodiment of the present application discloses a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the aforementioned scheduler-based SCA source code parsing method is implemented.
[0089] Embodiment 5
[0090] The embodiment of the present application discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of an electronic device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the electronic device executes the above-mentioned scheduler-based SCA source code parsing method.
[0091] It should be understood that in the embodiments of the present application, the processor referred to may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0092] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by hardware related to computer program instructions, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).
[0093] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of rights of the present invention. Therefore, equivalent changes made according to the scope of the patent application of the present invention are still within the scope covered by the present invention.
Claims
1. A scheduler-based SCA source code parsing method, characterized in that: The scheduler is used to establish collection nodes, computing nodes, migration nodes, merging nodes and abnormal nodes. The parsing methods include: Executing the collection node to collect a batch of component source code files of the components to be parsed, and starting the calculation node; Execute the computing node, parse the file hash of the component source code file, write it to Redis, and then start the migration node; Execute the migration node, migrate the file hash in Redis to Doris, and start the merge node; Execute the merge node to merge and remove duplicate file hashes in Doris, and return to start the computing node until the parsing of the component source code files of the current batch is completed, and then return to start the acquisition node to start processing a new batch of components to be parsed; When executing each node, the abnormal node is executed synchronously, and the execution of each node is monitored. If an abnormality occurs in the monitored node, a node retry mechanism is executed.
2. The method for SCA source code parsing based on a scheduler according to claim 1, characterized in that: The "collecting a batch of component source code files of components to be parsed" includes: According to the set number of components, multiple components to be parsed are grouped into a batch of components to be parsed; Download the component source code files of the batch of components to be parsed from the corresponding data source.
3. The method for SCA source code parsing based on a scheduler according to claim 1, characterized in that: The "parsing the file hash of the component source code file, writing it to Redis, and then starting the migration node" includes: Calculate a fragment hash according to the set number of lines as the file hash of the component source code file; Write the calculated file hash to Redis and deduplicate the same file hash in Redis; It is determined whether the memory of Redis reaches a set threshold. If so, the calculation of the fragment hash is suspended and the migration node is started. If not, the calculation of the fragment hash continues.
4. The method for SCA source code parsing based on a scheduler according to claim 1, characterized in that: The "migrating file hashes from Redis to Doris" includes: Use Flink to synchronize the file hash in Redis to the to-be-merged column in Doris; Clear the file hash in Redis.
5. The method for SCA source code parsing based on a scheduler according to claim 1, characterized in that: The "merging and deduplication of file hashes in Doris" includes: Use Doris's UDF function to merge the result column and the file hash in the column to be merged, and then deduplicate the data; The deduplicated file hash is saved to the result column, and the file hash in the to-be-merged column is cleared.
6. The method for SCA source code parsing based on a scheduler according to claim 1, characterized in that: The "node retry mechanism" includes: Re-execute the current node three times. If it fails three times, record the current execution result and send a notification message. The parsing of the component source code files of the current batch is terminated, and the collection node is returned to start the parsing of the component source code files of a new batch.
7. A SCA source code parsing device based on a scheduler, characterized in that: There are collection nodes, calculation nodes, migration nodes, merging nodes and abnormal nodes, and the analysis device includes: A collection module, used to execute the collection node, collect a batch of component source code files of the components to be parsed, and start the calculation node; A computing module, used to execute the computing node, parse the file hash of the component source code file, write it to Redis, and then start the migration node; Migration module, used to execute the migration node, migrate the file hash in Redis to Doris, and start the merge node; A merging module is used to execute the merging node, merge and de-duplicate the file hashes in Doris, and return to start the computing node or the collection node; The exception module is used to synchronously execute the abnormal node when executing each node, monitor the execution of each node, and execute the node retry mechanism if an abnormality occurs in the monitored node.
8. An electronic device, characterized in that: include: one or more processors; One or more memories are used to store one or more programs. When one or more of the programs are executed by the processor, the processor implements the scheduler-based SCA source code parsing method according to any one of claims 1 to 6.
9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the scheduler-based SCA source code parsing method according to any one of claims 1 to 6 is implemented.