An operating system-oriented open-source compliance efficient scanning method and system

Abstract

The application provides an open source compliance efficient scanning method and system for an operating system, belongs to the field of information technology, splits open source compliance scanning into a plurality of scanning items, realizes configurable differentiated scanning, judges a main task type after triggering a scanning task, obtains a scanning object according to the main task type, and reads a scanning template corresponding to the main task type, extracts metadata information of the scanning object, generates a uniform resource locator PURL, creates a subtask for the scanning item by using a distributed task scheduling framework, performs distributed subtask distribution based on execution cost calculation, and manages the scheduling and execution of the subtask. The application meets strict requirements for open source compliance in the operating system development, maintenance and supply chain management processes.

Description

Technical Field

[0001] This invention belongs to the field of information technology, specifically relating to an open-source, compliant, and efficient scanning method and system for operating systems. Background Technology

[0002] With the booming development and widespread application of open-source software, more and more enterprises and organizations are introducing and using open-source software in their project development processes. However, the use of open-source software also brings many compliance challenges and risks, requiring enterprises and organizations to effectively manage and control them.

[0003] The widespread use of open-source software has spurred demand for open-source compliance scanning tools. These tools are core technical means for enterprises to manage open-source compliance risks, providing systematic support from multiple dimensions such as legal compliance, supply chain security, development efficiency, cost control, and brand reputation.

[0004] Open-source compliance scanning tools typically include modules such as open-source code snippet referencing, dependency identification, license analysis, vulnerability detection, report generation, open-source knowledge base, and automated integration. A classic system architecture diagram is shown below. Figure 1 As shown in the diagram. The data acquisition layer serves as the "information entry point" for the open-source compliance scanning tool. Its core function is to efficiently and accurately acquire raw data from various sources, providing a solid foundation for subsequent compliance analysis. The analysis engine layer is the "intelligent hub" of the open-source compliance scanning tool. Its core function is to deeply process and logically manipulate the collected raw data, transforming it into interpretable compliance risk information. The data storage layer is the "information warehouse" of the open-source compliance scanning tool. Its core function is to systematically store, manage, and maintain the data generated during the collection and analysis process, ensuring data persistence, consistency, and traceability. The application presentation layer is the "interactive window" of the open-source compliance scanning tool. Its core function is to transform complex compliance scanning results into visualized and easily operable information, helping decision-makers, developers, auditors, and other roles efficiently complete the risk identification, handling, and management loop. The integration and extension layer is the "ecosystem connector" of the open-source compliance scanning tool. Its core function is to connect the data flow between the tool and external systems, achieving seamless integration of compliance management with the enterprise's existing technology stack, while also supporting the horizontal expansion of the tool's capabilities.

[0005] Currently, most open-source compliance scanning tools on the market are relatively uniform in their overall business processes, such as... Figure 2As shown, a typical compliance check includes core steps such as adding scan targets, preprocessing, executing the scan, and generating a report, but there are some differences in the specific implementation. Adding scan targets involves the user submitting the target to be scanned (such as code repositories, software packages, etc.) to the scanning tool and configuring relevant parameters (such as task type, detection content, etc.) to initialize the compliance check task. Preprocessing transforms the raw input into a standardized scan list by unpacking software packages / downloading code repositories / disassembling container images, filtering interfering files, and extracting basic metadata, providing a unified input format for the subsequent analysis engine. Executing the scan involves using a multi-dimensional analysis engine (dependency resolution, license identification, vulnerability scanning, etc.) to perform in-depth detection based on the preprocessing results, generating a standardized risk list and report. Generating a report is the process of converting the scan results into a structured document, providing users with a clear compliance assessment and decision-making basis by integrating violations, risk levels, and remediation suggestions.

[0006] Existing open-source compliance scanning tools primarily target code data from single projects, typically ranging from a few MB to hundreds of MB in size. However, when facing ultra-large-scale, multi-layered, and heterogeneous targets such as operating systems, these tools fall short. Operating systems possess inherent unique characteristics and complexity: they are not single projects but rather vast collections meticulously integrated from thousands of diverse and varied independent components, including kernels, system libraries, drivers, and applications. This means that a single compliance scan of an operating system involves an extremely large amount of code, often reaching hundreds of GB. This demand for ultra-large-scale, multi-layered, and heterogeneous scanning exceeds the design capacity of existing tools, exposing a series of problems:

[0007] 1. Insufficient adaptation to operating system scanning scenarios.

[0008] 1.1 Generalization of Heterogeneous Component Scanning Strategies: Operating systems consist of thousands of independent components of various forms and specifications. Existing tools adopt general scanning strategies and do not design differentiated scanning logic for the core audit points of different components, which often leads to problems such as "over-scanning" or "scanning blind spots" during the audit process.

[0009] 1.2 Weak support for software package formats: Existing tools do not perform accurate verification in accordance with the specifications of specific software package formats such as DEB and RPM, resulting in insufficient accuracy of compliance audit results and making it difficult to effectively support risk management across the entire operating system supply chain.

[0010] 1.3 Insufficient support for differentiated needs: Different product lines of operating systems (such as desktop, server, etc.), different development stages (such as initial inspection, release audit, etc.), different component types (such as kernel, application software, etc.), and different project attributes (such as open source projects, commercial projects, etc.) require different compliance strategies with different focuses. Existing scanning tools are difficult to adapt to the complex needs of multiple scenarios and multiple standards quickly and at low cost, and the scanning work lacks pertinence and practicality.

[0011] 2. Low scanning efficiency.

[0012] 2.1 Insufficient processing capacity for ultra-large-scale scans: Existing tools are mainly designed for single projects, and their architecture is ill-suited to handle ultra-large-scale scans of hundreds of GB of operating system code. Processing efficiency drops sharply with increasing scale, making it impossible to complete a full scan within a reasonable timeframe, severely impacting development iterations and version releases. Two domestic and two foreign tools of the same type were tested, and scanning 200 GB of operating system code took more than 24 hours in practice.

[0013] 2.2 The serial execution of scan items has a performance bottleneck: Although existing tools support the concurrent execution of multiple scan tasks, they cannot achieve the concurrent execution of multiple scan items for a single large scan object. They cannot make full use of the concurrency capabilities of multi-core CPUs, resulting in low utilization of computing resources and long scan cycles, which makes it difficult to meet the needs of CI / CD scenarios for rapid feedback.

[0014] 2.3 Frequent and Inefficient Knowledge Base Interactions: The detection process for core scanning items (such as open-source code snippet reference scanning, license detection, dependency component identification, and security vulnerability scanning) heavily relies on frequent and time-consuming interactions with large-scale open-source knowledge bases (such as open-source code repositories, license repositories, component repositories, and vulnerability databases). Existing tools lack effective caching and batch processing mechanisms. For modern operating systems containing tens of thousands of components, these frequent I / O operations and network requests have become a significant performance bottleneck restricting detection efficiency.

[0015] 2.4 Mandatory full execution of all scan items lacks flexibility: Open source compliance scanning covers many necessary scan items (such as project / component / file license identification, license conflict detection, license tampering detection, component / file copyright identification, copyright tampering detection, component identification, open source code snippet citation scanning, security vulnerability scanning, etc.), as well as unnecessary scan items (code self-development rate, programming language distribution, file type distribution, code line count statistics, etc.). Existing tools generally adopt a "one-size-fits-all" strategy, forcing the full execution of all scan items. However, different operating system product lines (such as desktop, server, etc.), different development stages (such as initial inspection, release audit, etc.), different component types (such as kernel, application software, etc.), and different project attributes (such as open source projects, commercial projects, etc.) have significantly different requirements for scan content. Existing technology cannot support users to specify scan items on demand, resulting in a large amount of unnecessary consumption of computing resources and wasted time.

[0016] 2.5 Lack of Historical Data Archiving Mechanism: Existing tools lack an effective historical data archiving mechanism, failing to archive and clean up historical scan task data in a timely manner. This leads to decreased database query performance and impacts the overall system response speed. Operating systems, due to their numerous components, frequent version iterations, and large number of developers, generate massive amounts of compliant scan data. Without archiving, core table data will continuously expand, far exceeding reasonable limits, significantly reducing query speed, easily causing system lag, and severely impacting scanning efficiency.

[0017] 2.6 Lack of a mechanism for reusing historical scan results: For the same scan target, regardless of whether different users submit scan requests multiple times or the same user initiates scan tasks repeatedly at different times, even if the scan items executed each time are exactly the same, existing tools will still mechanically repeat the scan process. While this problem may not have a significant impact on a single small-scale scan target, the lack of this mechanism will severely restrict scanning efficiency for operating systems with tens of thousands of components. Taking a certain version of an operating system as an example, which contains 4,000 components, if 2,000 of these components have already completed open-source compliance scanning, then when initiating a full scan of the operating system product later, the existing scan results of these components can be directly reused, skipping the repeated scanning process, thereby significantly improving scanning efficiency.

[0018] 2.7 Weak Interference File Filtering Capability: The weak interference file filtering capability of existing tools is one of the reasons for low scanning efficiency. On the one hand, filtering functions are lacking or the filtering dimensions are limited. Most existing tools either do not support interference file filtering, or even if they do, they only focus on filtering simple file types, failing to cope with the complex and diverse situation of interference files. A large number of irrelevant interference files are mixed into the scanning scope, increasing the workload. On the other hand, improper handling of compressed files either leads to direct skipping and missed scans, or unconditional decompression and recursive scanning, consuming a large amount of CPU, memory, and I / O resources, further slowing down the scanning speed. In addition, filtering rules are usually configured globally, unable to be flexibly adjusted and accurately applied according to the characteristics of different projects. This makes the scanning process unable to focus on the target code, ultimately resulting in long scanning time, high system load, and significantly low efficiency.

[0019] 3. Lacks flexibility and load balancing capabilities.

[0020] 3.1 Lack of distributed task scheduling capability: Most existing tools adopt a single-server architecture, and their processing capacity is limited by the hardware performance limit of a single server (such as CPU, memory, etc.). They cannot fully schedule and utilize the computing resources of multiple servers. When faced with large-scale scanning tasks, the resources of a single server are quickly exhausted, becoming an insurmountable performance bottleneck.

[0021] 3.2 Lack of elastic scaling capability: The current architecture cannot dynamically expand or shrink computing resources according to the task size, does not support linear scaling capability, and is prone to resource contention and performance degradation in high-concurrency scanning scenarios (such as before the release of the operating system).

[0022] 3.3 Lack of load balancing capability: Existing tools fail to comprehensively consider multiple factors such as differences in the size of scanned objects, significant differences in the time consumed by scanned items, and uneven server performance. They are unable to dynamically allocate scan tasks based on the real-time load of the server, resulting in some nodes being overloaded while others are idle, causing uneven resource utilization and ultimately affecting the overall throughput and stability of the system.

[0023] 4. The system's reliability and fault tolerance mechanisms are inadequate.

[0024] 4.1 Lack of Interruption Recovery Mechanism: Existing tools lack an effective task recovery mechanism when dealing with abnormal situations such as system power outages, crashes, or restarts. Interrupted scan tasks need to be manually recreated by the user, and the original task configuration and completed scan results cannot be retained, resulting in a large amount of repetitive work and wasted time.

[0025] 4.2 Single Point of Failure Risk: If a single node fails, the entire system will become unavailable, causing business continuity interruption. Such failures are not only costly to recover from, but also severely impact user experience. Summary of the Invention

[0026] To overcome the numerous challenges faced by existing open-source compliance scanning tools when scanning operating systems, this invention proposes an efficient open-source compliance scanning method and system for operating systems, which meets the stringent requirements for open-source compliance in the development, maintenance, and supply chain management of operating systems.

[0027] To achieve the above objectives, the technical solution of the present invention is as follows:

[0028] An open-source, compliance-efficient scanning method for operating systems includes:

[0029] S1. Break down open-source compliance scanning into several scanning items, each with its own independent scanning plugin; combine different scanning plugins using scanning templates to achieve configurable, differentiated scanning.

[0030] S2. After triggering the scanning task, determine the main task type, obtain the scanning object according to the main task type, and read the scanning template corresponding to the main task type;

[0031] S3. Extract the metadata information of the scanned object and generate a Uniform Resource Locator (PURL) as the identifier of the scanned object. Use the PURL to determine whether to execute the historical scan result reuse mechanism. If yes, stop the scanning task and directly return the corresponding historical scan result. Otherwise, proceed to step S4.

[0032] S4. A distributed task scheduling framework is adopted. Based on the scanning plugins configured in the scanning template corresponding to the main task type, subtasks are created for the scanning items corresponding to the scanning plugins. Distributed subtasks are dispatched based on execution cost calculation, and the scheduling and execution of subtasks are managed. The scanning results are summarized and displayed, and the scanning tasks and scanning results are stored as historical scanning data of successful execution.

[0033] Furthermore, in step S1, the relationship between the scanning template and the scanning plugin is maintained through three tables, including a template table that stores basic information of the scanning template, a plugin table that stores basic information of the scanning plugin, and a template-plugin relationship table that stores the relationship between the scanning template and the scanning plugin.

[0034] Furthermore, the triggering scan task in step S2 includes an active initiation mode and a passive triggering mode. The main task types of the active initiation mode include package scanning, code repository scanning, and operating system scanning. The main task types of the passive triggering mode include pipeline triggering, artifact warehousing triggering, and PR / MR triggering.

[0035] Furthermore, in step S2, after obtaining the scan object according to the main task type, interference file filtering is performed. The filtering methods include: excluding specified directories, excluding specified file types, scanning specified file types, and skipping specified compressed file types.

[0036] Furthermore, the method for generating the Uniform Resource Locator (PURL) in step S3 includes:

[0037] Depending on the package type of the software package or code repository, extract metadata information of the scanned object from the debian / control, debian / changelog, and debian / copyright files in the Debian package, or the .spec file in the RPM package, and the code repository URL string; generate a PURL that conforms to the specification based on the metadata information.

[0038] Furthermore, the method for determining whether to execute the historical scan result reuse mechanism in step S3 includes:

[0039] Based on the identifier PURL of the scanned object and the ID of the scan template corresponding to the main task type, query the successfully executed historical scan data to see if there are any historical scan tasks for the same scanned object and their corresponding historical scan results. If so, further determine whether the scan plugin in the scan template corresponding to the main task type is completely consistent with the plugin used in the historical scan task. If they are completely consistent, return the historical scan results directly.

[0040] Furthermore, the method of employing a distributed task scheduling framework described in step S4 includes:

[0041] Establish cloud-native deployment, perform distributed task scheduling, create a three-thread collaboration mechanism of Kafka thread, main thread, and patrol thread, and achieve fault tolerance through fine-grained management of task status.

[0042] Furthermore, methods for establishing cloud-native deployments include:

[0043] Standardized container packaging based on Docker;

[0044] Based on Kubernetes orchestration capabilities, differentiated deployment strategies are designed for the characteristics of different components, including a scheduling center, worker nodes, data services, and message middleware.

[0045] Introducing the Prometheus monitoring system, collecting dual-dimensional metrics of task load and node resources, and building custom elastic scaling rules;

[0046] By adopting GitOps continuous deployment, all Kubernetes resource inventories are stored in a Git repository, enabling versioned configuration management, automatic synchronization, and environment isolation.

[0047] Furthermore, the distributed task scheduling includes:

[0048] Construct a two-level task model that includes main tasks and sub-tasks;

[0049] The distributed task dispatch based on execution cost is calculated using the following formula: Where C represents the execution cost of the scanning task at a certain working node; S represents the size of the scanned object; n represents the number of scanned items, and i represents the i-th scanned item; The time consumption weight of the i-th scan item is pre-set based on the computational complexity and resource consumption of the scan item; P represents the performance index of the worker node, calculated based on the number of CPU cores, memory size, and throughput.

[0050] Each worker node creates a Kafka thread, a main thread, and a patrol thread upon startup, and completes the scanning task through a three-thread collaborative mechanism.

[0051] In another aspect, this invention proposes an open-source, compliant, and efficient scanning system for operating systems, comprising:

[0052] Module splitting and combination: Open source compliance scanning is broken down into several scanning items, each of which is implemented by an independent scanning plugin; different scanning plugins can be combined through scanning templates to achieve configurable differentiated scanning;

[0053] Trigger Scan Module: After triggering a scan task, determine the main task type, obtain the scan object based on the main task type, and read the scan template corresponding to the main task type;

[0054] Reuse Judgment Module: Extracts metadata information of scanned objects, generates Uniform Resource Locator (PURL) as the identifier of scanned objects, and uses the PURL to determine whether to execute the historical scan result reuse mechanism. If so, the scanning task is stopped and the corresponding historical scan result is returned directly; otherwise, it enters the distributed task module.

[0055] Distributed Task Module: Employs a distributed task scheduling framework. Based on the scanning plugins configured in the scanning template corresponding to the main task type, it creates subtasks for the scanning items corresponding to the scanning plugins, performs distributed subtask dispatch based on execution cost calculation, and manages the scheduling and execution of subtasks. It summarizes and displays the scanning results and stores the scanning tasks and results as historical scanning data of successful execution.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] (1) The scanning efficiency is greatly improved: The present invention improves the scanning efficiency several times by adopting a distributed architecture design, introducing a parallel execution mechanism, improving the knowledge base interaction method, providing the ability to configure scanning items on demand, establishing a historical data migration mechanism, introducing a historical scanning result reuse mechanism, and designing an interference file filtering algorithm.

[0058] (2) Achieved elastic scaling and load balancing: This invention uses cloud-native elastic scaling rules based on custom metrics, enabling the system to dynamically expand or shrink computing resources according to task size, balancing efficiency and economy. At the same time, the scheduling center adopts a distributed task dispatch algorithm based on execution cost calculation to achieve efficient utilization of cluster resources and improve the overall throughput of the system.

[0059] (3) The system reliability is significantly enhanced: With the help of a sound fault tolerance and interruption recovery mechanism, the present invention can effectively deal with abnormal situations such as node failure and task timeout, ensure business continuity, eliminate single point of failure, and meet the enterprise-level high availability requirements.

[0060] (4) Strong scene adaptability and flexibility: In view of the special nature and complexity of the operating system, this invention designs differentiated scanning strategies for different components, and combines the specification requirements of specific software package formats such as DEB and RPM to carry out accurate verification, quickly and cost-effectively adapt to the diverse compliance policy requirements of different operating system product lines and versions, making the scanning work more targeted and practical. Attached Figure Description

[0061] Figure 1 This is a schematic diagram of the system architecture of existing open-source compliance scanning tools;

[0062] Figure 2 This is a business process diagram of an open-source compliance scanning tool in existing technology;

[0063] Figure 3 This is a flowchart of Embodiment 1 of the present invention. Detailed Implementation

[0064] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0065] Example 1:

[0066] The technical problems to be solved by this invention include: enhancing adaptability to operating system scanning scenarios, improving scanning efficiency, achieving elastic scaling and load balancing, and improving system reliability and fault tolerance mechanisms.

[0067] Based on the above problems, this embodiment proposes an open-source, compliant, and efficient scanning method for operating systems, such as... Figure 3 As shown, it includes:

[0068] I. Multi-scenario scanning trigger mechanism.

[0069] To address the diverse needs of "actively initiated scanning" and "passively triggered scanning" during the operating system development process, the system is designed with a multi-scenario scanning trigger mechanism to achieve deep integration with the entire operating system development process and ensure comprehensive compliance scanning of the operating system.

[0070] 1.1 Proactive Initiation Mode:

[0071] Users can initiate scanning tasks directly through the system's front-end interface. It supports both full scans of operating system products, covering all components for comprehensive compliance testing, and individual scans of specific components within the operating system product, meeting the needs of precise location and on-demand testing scenarios.

[0072] 1.1.1 Package Scanning: Supports local file upload (maximum single file size 10GB) and remote resource download (compatible with HTTPS / HTTP / FTP protocols). The system has a built-in intelligent decompression engine that can automatically identify common compressed file formats such as ZIP, RAR, and 7Z without manual intervention.

[0073] 1.1.2 Code Repository Scanning: Adapted to mainstream version control systems such as Git and SVN, users can flexibly configure target branches / tags and access credentials (SSH key or username and password) after entering the repository address. The system supports "full pull" and "shallow clone optimization" (pulling only the latest version of the specified branch), balancing scan completeness and efficiency.

[0074] 1.1.3 Operating System Scan: Supports selecting specific operating system products (such as desktop operating system V1.0, server operating system V1.0, etc.), and the system automatically obtains the package list from the artifact hosting platform (such as Artifactory, koji, KylinBuilder, etc.) to achieve a full operating system-level scan.

[0075] 1.2 Passive Trigger Mode:

[0076] By deeply integrating with standardized API interfaces and external R&D toolchains, and establishing a complete closed loop of event triggering, automatic scanning, and real-time feedback processes, manual intervention is significantly reduced. This ensures that compliance checks are seamlessly embedded in key nodes of the R&D process, achieving simultaneous progress in compliance control and R&D. In this embodiment, passive triggers include pipeline triggers, product warehousing triggers, and PR / MR triggers. Figure 3 The section on passive triggering only shows scans caused by PR / MR triggering.

[0077] 1.2.1 Pipeline Trigger: Integrate with CI / CD tools (such as Jenkins, GitLabCI, etc.) to call the scanning API in the pipeline script to realize the automated process of code submission -> automatic build -> compliance scan.

[0078] 1.2.2 Product Inbound Trigger: Connects to product repositories such as Nexus and Harbor, automatically triggering a scanning task when a new software package is inbound, intercepting software packages with compliance risks, and preventing them from entering the product repository.

[0079] 1.2.3 PR / MR Trigger: Integrated with code hosting platforms such as GitLab, GitHub, and Gitee, when developers create a Pull Request (PR) or Merge Request (MR), the system automatically extracts the changed files (incremental code), triggers an incremental scan, and the scan results are fed back to the PR / MR page in real time.

[0080] II. Scanning template configuration mechanism.

[0081] To meet the personalized scanning needs of different business scenarios, the system adopts an architecture design of "plug-in splitting + template configuration" to achieve flexible configuration and dynamic adjustment of scanning strategies. It completely decouples the "scheduling framework" and "business logic" from the underlying architecture level, which not only greatly improves the scanning efficiency of the system, but also ensures the accurate adaptation capability of the scanning strategy.

[0082] 2.1 Plug-in Decomposition:

[0083] 1) Break down open-source compliance scanning into several scanning items, with each scanning item having its functionality implemented by an independent plugin;

[0084] 2) When initiating a scan, a subtask is created for each scan item, and the task scheduling framework manages and executes them uniformly.

[0085] 3) Fully utilize the concurrency capabilities of multi-core CPUs to support concurrent execution of subtasks to further improve scanning efficiency.

[0086] 2.2 Templated Configuration:

[0087] By combining different scanning plugins with scanning templates, flexible and configurable differentiated scanning can be achieved to meet the needs of different operating system product lines (such as desktops and servers), different development stages (such as initial inspection and release auditing), different component types (such as kernels and application software), and different project attributes (such as open-source projects and commercial projects), allowing for customized and differentiated scanning strategies. The relationship between templates and plugins is maintained through three tables:

[0088] 1) Template table: Used to store basic information about scanned templates. Fields include template name, template description, applicable scenarios, creator, creation time, and whether it is the default template.

[0089] 2) Plugin table: Used to store basic information about scanning plugins. Fields include plugin name, plugin description, Python function name, function parameters, plugin version, and whether it is enabled.

[0090] 3) Template and plugin relationship table (template_plugin): Used to maintain the one-to-many relationship between scanning templates and scanning plugins. Fields include template ID and plugin ID, etc.

[0091] III. Standardized pretreatment production line.

[0092] To address the diverse forms and formats of objects scanned in operating system scenarios, a unified preprocessing workflow was designed to output a standardized "scan list," providing consistent and standardized input data for the subsequent analysis engine and avoiding scanning deviations caused by differences in data formats.

[0093] 3.1 Obtaining Scanned Objects:

[0094] 3.1.1 Package Acquisition. Supports local file upload and remote resource download.

[0095] 3.1.2 Code repository retrieval. Supports Git pull and SVN checkout.

[0096] 3.1.3 Product Acquisition. Supports obtaining a list of product software packages from the hosting platform.

[0097] 3.2 Interference File Filtering:

[0098] For interference files, this embodiment designs a set of efficient and comprehensive filtering algorithms, which support targeted filtering of interference files from the following four dimensions.

[0099] 3.2.1 Exclude Specified Directories: By configuring a comma-separated list of directories, directories within the list and all their contained files and subdirectories will be skipped from the scan. Common directories to exclude include, but are not limited to:

[0100] Dependency management directories: bower_components, vendor, jspm_packages, Carthage, Pods, .gradle, deps, _build, .m2, .npm, node_modules;

[0101] Build output directories: build,target,dist,bin,obj,out,debug,.next,.nuxt,.svelte-kit,lib;

[0102] Development environment directories: .pyenv, env, venv, ENV, .venv;

[0103] Cache directories: .cache, .pytest_cache, .nyc_output, .parcel-cache, pycache;

[0104] Document directory: docs;

[0105] IDE configuration directories: .idea, .vs, .vscode-test, .vscode;

[0106] 3.2.2 Exclude Specified File Types: By configuring a comma-separated list of file extensions, files with extensions in this list will not be scanned. Common file types to exclude include, but are not limited to:

[0107] Compressed / archived files: zip, rar, tar, 7z, etc.

[0108] Video files: mp4, avi, mov, wmv, mkv, etc.

[0109] Image files: jpg, jpeg, png, gif, webp, bmp, tiff, tif, psd, ico, heic, heif, svg, ai, eps, etc.

[0110] Document files: chm, pdf, etc.

[0111] Executable files: exe, etc.

[0112] 3.2.3 Scan specified file types: By configuring a comma-separated list of file extensions, only files with extensions in that list will be scanned.

[0113] Specify source file types: c, h, cpp, java, go, py, js, cs, ts, php, pl, PL, rs, swift, kt, R, rb, dart, f, for, m, scala, erl, groovy, ada, html, css, vue, less, json, etc.

[0114] 3.2.4 Skip specified compressed file types: By configuring a comma-separated list of compressed file extensions, decompression operations will not be performed on files corresponding to the extensions in the list, and there is no need to perform recursive scanning on the decompressed contents.

[0115] Common compressed file types: zip, rar, 7z, war, tar, gz, xz, tgz, etc.

[0116] The system provides a configuration page for filtering interfering files and supports binding it to a scanning template to ensure that the rules only apply to tasks created based on that template.

[0117] 3.3 Source Data Extraction:

[0118] Metadata information of the scanned object is extracted from the license file in the root directory of the package or code repository, the debian / control, debian / changelog, and debian / copyright files in Debian packages, the .spec file in RPM packages, and the code repository URL string. The extracted metadata includes the package / code repository name, description, summary, version number, release number, package type, project license, product, vendor, homepage link, and download address. Based on this information, a compliant PURL (PackageURL) is generated as the unique identifier for the package.

[0119] 3.4 Metadata storage;

[0120] To achieve efficient management of metadata for scanned objects, a Package table was designed to comprehensively record information on all software packages and code repositories scanned by the system throughout history. This table uses PURL (PackageURL) as the unique identifier for each software package / code repository. Leveraging the standardized nature of PURL generation based on metadata (such as name, type, and version number), even if the same software package / code repository is submitted multiple times by different users, or if the same user initiates multiple scan tasks at different times, the system can accurately match and identify it as the same scanned object using PURL.

[0121] For scenarios identified as duplicate scans, the system will automatically associate the historical scan results corresponding to the PURL and reuse them directly without re-executing the scan analysis process. This avoids redundant calculations at the source, significantly improving the overall scan response efficiency while effectively reducing the consumption of server computing power, storage and other resources.

[0122] IV. Distributed Task Scheduling Framework.

[0123] This framework adopts an architecture concept of "centralized scheduling + distributed execution," deeply integrating cloud-native technologies to address the challenges of efficiency, reliability, and elastic scaling faced by operating systems in ultra-large-scale scanning scenarios. The framework encompasses four core designs: cloud-native deployment, distributed task scheduling, high reliability and fault tolerance, and three-thread collaboration, which together construct a complete and automated task scheduling and execution system.

[0124] 4.1 Cloud-native deployment:

[0125] 4.1.1 Containerized packaging: All core components (scheduling center, worker nodes, front-end services, database services, Kafka, etc.) are packaged in standardized containers based on Docker, strictly adhering to the cloud-native concept of "build once, run anywhere", which greatly simplifies the deployment process and improves deployment efficiency.

[0126] 4.1.2 Kubernetes Orchestration: Based on Kubernetes orchestration capabilities, differentiated deployment strategies are designed for the characteristics of different components to balance high availability and resource utilization.

[0127] 1) Scheduler: Deployed using a Deployment controller, with a default configuration of 1.

[0128] Configuration management adopts a "separation of sensitive and non-sensitive information" approach:

[0129] Non-sensitive configurations such as database connection strings and Kafka cluster addresses can be injected via ConfigMap, allowing for online modification and real-time effect.

[0130] Sensitive information such as API access keys and knowledge base access tokens are stored using Secret encryption to prevent plaintext disclosure of configurations, which complies with data security standards.

[0131] 2) Worker: Also deployed using the Deployment controller, with an initial number of 2 replicas to meet daily scanning needs. It can automatically expand or shrink according to the workload to avoid idle or overloaded resources.

[0132] 3) Data service: MySQL is deployed using a StatefulSet controller, which mounts PersistentVolume (PV) to achieve data persistence. At the same time, a master-slave replication architecture is configured to ensure that data is not lost.

[0133] 4) Message middleware: Kafka is deployed in a two-node cluster, which uses a "partition replica" strategy to ensure the reliable delivery of task messages (two replicas per partition by default), avoids message loss due to single-node failure, and supports the stable flow of task scheduling.

[0134] 4.1.3 Elastic Scaling Based on Custom Metrics: Traditional elastic scaling strategies based on CPU / memory utilization cannot accurately reflect the load characteristics of scanning tasks (e.g., task queue backlog may occur before CPU overload). To address this, the framework introduces the Prometheus monitoring system to collect dual-dimensional metrics of "task load + node resources," constructing custom elastic scaling rules to achieve precise resource and load matching.

[0135] 1) Trigger indicator:

[0136] Task queue length: directly reflects the task backlog and is the core basis for scaling decisions;

[0137] Average task processing time: This reflects the current cluster processing efficiency; an increase in processing time indicates an increase in load.

[0138] Worker node CPU / memory utilization: helps determine whether node resources are overloaded, avoiding node crashes caused by scaling based solely on task volume.

[0139] 2) Scaling strategy:

[0140] Expansion trigger: When the task queue length of all worker nodes, CPU / memory usage exceed the preset warning value, and the average task processing time shows an upward trend, the number of worker node replicas will be automatically increased.

[0141] Scaling down triggers: When the task queue length, CPU / memory utilization of all worker nodes are lower than the preset safety value, and the average task processing time shows a downward trend, the number of worker node replicas is automatically reduced.

[0142] 3) Capacity expansion / contraction protection:

[0143] Cooldown time setting: Set the scaling cooldown time, which means that after a scaling operation is triggered, no new scaling requests will be responded to within a specified time, so as to avoid "frequent scaling" caused by high frequency fluctuations in task load and reduce the impact of resource jitter on business.

[0144] Cold start optimization: The cold start logic of worker nodes is optimized. Commonly used resources (such as plugin cache, basic scanning rule base, etc.) are automatically loaded during the node startup phase, reducing the cold start time from the traditional 2-3 minutes to within 30 seconds. This ensures that newly added nodes can be quickly connected to the task processing flow after expansion, improving load handling efficiency.

[0145] 4.1.4 GitOps Continuous Deployment.

[0146] 1) Configuration storage: All Kubernetes resource manifests (Deployment / YAML, ConfigMap) are stored in a Git repository (such as GitLab) to achieve versioned configuration management.

[0147] 2) Automatic synchronization: ArgoCD monitors Git repository changes and automatically synchronizes them to the Kubernetes cluster when the configuration is updated. The synchronization process includes health checks (such as Pod startup status and service availability).

[0148] 3) Environment isolation; development / testing / production environments are distinguished by namespaces, and different environments use independent configurations (such as lower elastic scaling thresholds for testing environments) to ensure deployment security.

[0149] 4.2 Distributed Task Scheduling:

[0150] 4.2.1 Two-level task structure: Construct a two-level task model of "main task - sub-task" to realize refined management and parallel execution of tasks.

[0151] 1) Main task: Corresponds to the complete scan request. It is a holistic description of the scan task and provides a basis for subsequent task splitting and allocation. Specifically, it includes the task name, task type, task status, scan template, scan object information, and local storage path of the scan object.

[0152] 2) Subtasks: Subtasks are derived from main tasks. Each subtask corresponds to the execution flow of a single scanning plugin. The parent-child relationship is recorded through a task relationship table, which realizes clear management of task hierarchy and facilitates task tracking and monitoring.

[0153] 4.2.2 Task Dispatch: In practical applications of open-source compliance scanning, the size of the scanned objects varies significantly, typically ranging from several MB to several GB. Even for the same scanned object, the execution time for different scan items varies. For example, open-source code fragment reference scanning is usually the most time-consuming, while project license identification is usually faster. Furthermore, for the same scanned object, the time required to execute the same scan item may also differ due to performance differences in worker nodes.

[0154] Given the unique nature of operating system open-source compliance scanning, this invention designs a distributed task dispatch algorithm based on execution cost calculation, the calculation formula of which is as follows:

[0155] ;

[0156] The parameters are defined as follows:

[0157] C (Cost): Represents the execution cost of the scan task at a certain working node;

[0158] S (Size): Represents the size of the object being scanned;

[0159] n represents the number of items scanned;

[0160] i represents the i-th scan item;

[0161] W (Weight): The time consumption weight of each scan item, which is preset according to the computational complexity and resource consumption of different scan items;

[0162] The time consumption weight of the i-th scan item;

[0163] P (Performance): Represents the performance metric of a worker node, calculated based on the number of CPU cores, memory size, and throughput.

[0164] The system maintains a queue of tasks to be processed for each worker node and dynamically calculates the total execution cost for each task queue. When a new scan task is created, the system performs the following two core operations:

[0165] 1) Based on the above formula, calculate the execution cost of the task at each working node;

[0166] 2) Simulate the scenario of dispatching new tasks to each worker node, recalculate the total execution cost of the task queue to be processed on each worker node after dispatch, and calculate the variance based on these total execution costs. The smaller the variance, the more balanced the load distribution of each worker node and the higher the overall system resource utilization.

[0167] Ultimately, the system selects the task dispatch scheme that minimizes this variance, assigning scanning tasks to the corresponding worker nodes. Once a worker node removes a task from the task queue, the system immediately recalculates and updates the total execution cost of that task queue. This algorithm effectively prioritizes dispatching larger, more time-consuming scanning tasks to more powerful worker nodes, thereby improving overall task execution efficiency and resource utilization.

[0168] 4.2.3 Task Execution: Each worker node creates a Kafka thread, a main thread, and a patrol thread upon startup. The Kafka thread periodically retrieves tasks to be processed from the worker node's task queue, creates a main scan task, initializes its status to Pending, and sets its due date to the current time.

[0169] After the main thread retrieves the expired main task, it first performs preprocessing operations, including object extraction, package decompression, package format verification, interference file filtering, and metadata extraction and saving. Based on the extracted metadata, a PURL conforming to the specifications is generated. Then, the Package table is queried to check if it already exists. If it does, it is determined to be a duplicate scan, and the system will automatically associate the historical scan results corresponding to that PURL and reuse them directly.

[0170] After preprocessing, the main task first reads the scan template configuration information to obtain a list of scan plugins to be executed. Then, it assembles subtask information for each scan plugin and creates subtasks one by one. The status of each subtask is initially initialized to Pending, the expiration time is set to the current time, and the parent-child relationship between the subtask and the main task is saved.

[0171] The main thread periodically retrieves a certain number of due subtasks from the task table. The specific execution logic of these subtasks is determined by the function and params fields in the plugin table, and all subtasks will be executed concurrently.

[0172] During the execution of subtasks, a caching strategy is implemented for frequently accessed knowledge base data such as license libraries, component libraries, and vulnerability libraries to reduce the number of network requests.

[0173] The inspection thread continuously tracks the execution progress of all subtasks associated with the main task. Once all subtasks are completed, it automatically initiates result aggregation, collecting and integrating subtask analysis results to generate a standardized compliance report. The inspection thread also triggers a scheduled archiving task daily, migrating records older than 30 days from the task table to the history table and freeing up storage space in the main table, ensuring long-term efficient system operation.

[0174] 4.3 High reliability and fault tolerance:

[0175] 4.3.1 Refined management of task status.

[0176] 1) Task status definition:

[0177] Clearly define all states throughout the entire task lifecycle, covering all stages of execution, and ensure that the state semantics are clear and unambiguous.

[0178] The system defines the following task states: Pending, Running, Completed, Failed, and Timeout.

[0179] 2) State persistence:

[0180] All changes to the task status are written to the task table in real time, recording key information such as execution node and change time, to ensure that the status is traceable and auditable.

[0181] 4.3.2 Automatic fault detection and recovery.

[0182] 1) Node Failure Detection: The scheduling center monitors the status of worker nodes in real time by integrating the Kubernetes API. When a node is detected to be down, unresponsive, or experiencing service abnormalities for more than 1 minute, the node is automatically marked as an "abnormal node" and the subsequent recovery process is triggered.

[0183] 2) Reassignment of tasks for abnormal nodes: The scheduling center retrieves the tasks to be processed from the task queue of abnormal nodes and reassigns them to the execution nodes based on the resource load of the remaining normal nodes to ensure that the tasks are not interrupted.

[0184] 3) Subtask Resume: For subtasks that need to be executed in stages, the system supports real-time storage of intermediate results. When a subtask is interrupted due to node failure, network interruption, or other reasons, the stored intermediate results can be directly read when execution resumes, and execution can continue from the breakpoint, avoiding a full re-execution and improving efficiency.

[0185] 4.3.3 Task fault tolerance mechanism.

[0186] 1) Automatic retry: An automatic retry mechanism is triggered by default after a task fails, with a preset number of retries of 3. To avoid repeated failures due to momentary resource conflicts or network fluctuations, the retry interval adopts an exponentially increasing strategy, that is, the first retry interval is 2 minutes, the second interval is 4 minutes, and the third interval is 8 minutes, reducing the concentrated occupation of system resources.

[0187] 2) Timeout control: The system supports configuring timeout time for scanning tasks. When a scanning task has not been completed by the timeout period, its status will be automatically marked as Timeout and an alarm notification (such as SMS, email, or maintenance platform alarm) will be triggered simultaneously to remind maintenance personnel to investigate the cause of the timeout in a timely manner.

[0188] 3) Overload protection: Based on the hardware resources of the working nodes, the maximum number of concurrent tasks per node is set to effectively prevent resource contention and node overload caused by too many tasks, thus ensuring the overall stability and performance of the system.

[0189] 4.4 Three-thread collaboration:

[0190] 4.4.1 Kafka Threads: Kafka threads can periodically query the number of scan tasks currently being executed in the current worker node. When the number is less than the maximum number of concurrent tasks, several tasks to be processed are taken from the worker node's task queue, and scan tasks are actually created. The status is initialized to Pending, and the task expiration time is set to the current time.

[0191] 4.4.2 Main thread: Periodically retrieves a certain number of expired tasks with a status of Pending from the task table, creates corresponding subtasks in sequence according to the scanning plugins configured in the scanning template, and saves the parent-child relationship between tasks.

[0192] 4.4.3 Patrol Thread:

[0193] 1) Status monitoring: The task table is periodically scanned every hour to check the update time of Running tasks. If it has not been updated for more than 1 hour, it is regarded as a "timeout task" and the status is updated to Timeout.

[0194] 2) Results Summary: Tracks the execution progress of all subtasks associated with the main task. When all subtasks are completed, the results summary is automatically started. It collects and integrates the subtask analysis results, generates a standardized compliance report, and provides a complete basis for decision-making.

[0195] 3) Historical Archiving: A scheduled archiving task is triggered at 2:00 AM every day to migrate records older than 30 days from the task table to the historical table and release storage space in the main table to ensure long-term efficient operation of the system.

[0196] Through the above design, the system achieves full automation from scan triggering to result generation, can efficiently handle ultra-large-scale scanning tasks at the operating system level, and has flexible configuration capabilities, reliable fault tolerance mechanisms and elastic resource scaling capabilities, meeting the open source compliance management requirements of the entire life cycle of the operating system.

[0197] V. Key Technologies

[0198] The key protection points in the method proposed in this embodiment are explained as follows:

[0199] 5.1 Scanning architecture based on plug-in-based splitting and template-based configuration:

[0200] 1) Core innovation: Through the structured design of three core tables - template table - plugin table - template and plugin relationship table, the scanning function is broken down into several scanning items. Each scanning item is implemented by a separate scanning plugin. Different scanning plugins are combined through scanning templates, thus completely decoupling the scheduling framework and business logic at the architectural level.

[0201] 2) Technical Differentiation: Breaking through the rigid limitation of existing tools that force all scan items, it supports users to customize differentiated scanning strategies according to actual needs, for different product lines of operating systems, different development stages, different component types, and different project attributes.

[0202] 3) Patent value: This architecture enables plug-and-play scanning functionality. When adding a new scanning item, only the corresponding scanning plugin needs to be developed and the function and params fields need to be configured in the plugin table. There is no need to modify the core framework code, which greatly reduces the difficulty and cost of functional expansion, significantly improves the scalability of the system, and can quickly respond to ever-changing scanning needs.

[0203] 5.2 Multi-dimensional interference file filtering algorithm:

[0204] 1) Core Innovation: A highly efficient and comprehensive interference file filtering algorithm was designed, achieving targeted filtering of interference files from four dimensions: excluding specified directories, excluding specified file types, scanning specified file types, and skipping specified compressed file types. Compared to single-dimensional filtering methods, this multi-dimensional comprehensive filtering approach can more accurately and flexibly meet the filtering needs of interference files in different scenarios, effectively improving filtering efficiency and accuracy.

[0205] 2) Technical Differences: Compared with traditional scanning tools that do not support filtering of interfering files or only support filtering of simple file types, this solution has the following significant differences:

[0206] Balancing comprehensiveness and flexibility: Unlike existing technologies that do not focus on interfering files or only focus on a single dimension, this invention considers multiple dimensions such as directories and file types (including ordinary files and compressed files), achieving comprehensive and flexible filtering.

[0207] Unique feature of compressed file processing: For compressed files, this invention skips a specified list of compressed file types by configuration, and directly avoids decompressing files with extensions in the list and subsequent recursive scanning.

[0208] Precision of rule application: This invention uses a technique that binds filtering rules to scanning templates, unlike traditional methods where filtering rules may be globally effective or difficult to apply precisely to specific tasks. This binding mechanism ensures that filtering rules apply precisely to tasks created based on the corresponding templates, allowing different tasks to use different filtering rules according to their needs, thus improving the precision of rule application and the system's adaptability.

[0209] 3) Patent Value: This algorithm effectively reduces the number and scope of files scanned by the system, lowers system resource consumption, and improves scanning speed and efficiency, thereby enhancing the overall system performance. The configuration method, which is bound to the scanning template, allows the system to flexibly customize filtering rules according to different business needs and scenarios, and facilitates the management and maintenance of these rules.

[0210] 5.3 PURL-based algorithm for reusing historical scan results:

[0211] The Uniform Resource Locator (PURL) specification defines a lightweight, universal syntax for identifying software packages. By adopting a URL-based format, PURL provides a consistent and interoperable mechanism across various ecosystems and tools. It is designed to address the ambiguity, inconsistency, and fragmentation inherent in package identification, offering better interoperability and traceability for the modern software supply chain.

[0212] PURL addresses the ambiguity of package identification, interoperability across ecosystems, enhances package traceability and risk management capabilities, and simplifies tool development, automation, and integration processes.

[0213] PURL consists of seven parts: scheme, type, namespace, name, version, qualifiers, and subpath.

[0214] When the same scan item is triggered multiple times for the same scan object, the system will directly reuse the historical scan results to improve scanning efficiency and reduce resource consumption.

[0215] First, depending on the package type of the software package or code repository, metadata information of the scanned object is extracted from the debian / control, debian / changelog, and debian / copyright files in Debian packages, or the .spec file in RPM packages, and the code repository URL string. The extracted metadata includes information such as the package / code repository name, version number, package type, and namespace. Based on this information, a standardized PURL (PackageURL) is generated as the unique identifier for that software package.

[0216] Then, based on the PURL of the scanned object and the scan template ID, it checks whether a successfully executed scan task exists. If it exists, it further determines whether the scan plugin in the current scan template is completely identical to the plugin used in the historical task. If they are completely identical, the historical scan result is returned directly.

[0217] 5.4 Distributed task dispatch algorithm based on execution cost calculation:

[0218] 1) Core Innovation: By comprehensively considering key factors such as the size of the scanned object, the time weight of each scan item, and the performance of the worker nodes, an execution cost calculation model is constructed to achieve accurate quantification of task execution costs. First, the system maintains a queue of tasks to be processed for each worker node and dynamically calculates the total execution cost of each task queue. Then, when a new scan task is created, the execution cost of that task on each worker node is calculated separately, and the scenario of dispatching the new task to each worker node is simulated, recalculating the total execution cost of the queues of tasks to be processed on each worker node after dispatch. Finally, based on these total execution costs, the variance is calculated, and the system selects the task dispatch scheme that minimizes this variance, dispatching the scan task to the corresponding worker node.

[0219] 2) Technical Differentiation: Unlike traditional load balancing strategies such as round-robin, weighted round-robin, or minimum load, this algorithm fully considers the special characteristics of open-source compliant scanning scenarios. It comprehensively considers various factors such as large differences in the size of scanned objects, significant variations in scan item execution time, and uneven node performance, overcoming the limitations of complex scanning scenarios. It achieves dynamic load balancing through a variance minimization strategy, rather than simply pursuing the resource utilization of a single node. Innovatively, it correlates task execution cost with node performance, enabling intelligent allocation of large tasks to high-performance nodes and reasonable distribution of small tasks to lightly loaded nodes. This solves the problems of resource allocation imbalance and low concurrency efficiency in traditional scheduling methods.

[0220] 3) Patent value:

[0221] Significantly improves resource utilization: The algorithm can accurately allocate large-scale, time-consuming tasks to high-performance nodes, while reasonably arranging small-volume, fast-execution tasks, so that system resources are fully utilized.

[0222] Significantly shorten the scanning cycle: By optimizing the task dispatch strategy, efficient matching of tasks and node resources is achieved, reducing task waiting time and execution time.

[0223] Enhanced system adaptability and stability: This algorithm has dynamic load awareness and adaptive scheduling capabilities, and can automatically adjust the task dispatch scheme according to the real-time status of the system and task characteristics.

[0224] Forming a differentiated competitive advantage through technology: Unlike the static task allocation or simple polling strategies of existing technologies, this algorithm forms a unique dynamic scheduling mechanism, providing an innovative solution for the field of open-source compliance scanning of operating systems.

[0225] 5.5 Cloud-native elastic scaling algorithm based on custom metrics:

[0226] 1) Core Innovation: To address the issue that traditional elastic scaling strategies based on CPU / memory utilization cannot accurately reflect the characteristics of scan task load, the system introduces the Prometheus monitoring system to collect dual-dimensional indicators of task load and node resources, build custom elastic scaling rules, and achieve precise adaptation of resources and load.

[0227] 2) Technical differences: Existing tools are mostly single-server architectures or fixed cluster sizes, which cannot cope with the tidal and wave-like scanning needs in the operating system field. This invention uses task queue length, average task processing time and worker node CPU / memory utilization as elastic scaling trigger indicators, sets scaling cooldown time, and optimizes worker node cold start logic to achieve dynamic matching of resources and load.

[0228] 3) Patent value: This algorithm enables the system to expand exponentially (2->4->8) working nodes under high load (such as full operating system scan) to meet the requirement of completing 200GB code scan within 6 hours; under low load, it can shrink to 2 working nodes, reducing resource costs by 75%, balancing scanning efficiency and cost control. This is the key technology of this invention for adapting to cloud-native environments and supporting ultra-large-scale scanning.

[0229] 5.6 Four-Table, Three-Thread Task Scheduling Framework:

[0230] The core data tables include: job (master / subtask information), job_relation (task relationship), job_his (historical master / subtask information), and job_relation_his (historical task relationship).

[0231] The core threads include:

[0232] Kafka threads:

[0233] 1) Consume task data from the Kafka queue and insert it into the job table; 2) Shorten the foreground response time through the message middleware (Kafka).

[0234] Main thread:

[0235] 1) Retrieve and execute expired tasks on a regular schedule; 2) Flow control mechanism to prevent server overload; 3) Automatically handle parent-child relationships between tasks.

[0236] Inspection Thread:

[0237] The triple responsibility design includes: 1) checking the update time of Running tasks. If the update is not completed within the specified time, the task is considered to have expired and its status is updated to Timeout; 2) polling the status of subtasks → summarizing the results / updating the main task / generating a report; 3) periodically migrating expired tasks to the historical table to control the size of the core table, thereby significantly improving the framework's performance and maintainability and avoiding data bloat in the core table.

[0238] 5.7 Cloud-native distributed task scheduling:

[0239] Containerization and Kubernetes orchestration: Core components such as the scheduling center and worker nodes all provide Docker container images, which can be deployed on Kubernetes clusters. The YAML manifest file enables automated deployment, rolling updates and self-healing management of services.

[0240] GitOps continuous deployment: The system deployment manifest is versioned in the Git repository, and combined with tools such as ArgoCD, it realizes declarative deployment and automated synchronization, ensuring environment consistency and improving deployment efficiency and auditability.

[0241] Cloud-native configuration management: Utilize Kubernetes ConfigMap and Secret to uniformly manage application configuration and sensitive information, achieving separation of configuration and application, enhancing security and ease of cross-environment deployment.

[0242] Centralized scheduling-distributed execution architecture: The system adopts a distributed architecture of a scheduling center (Scheduler) + worker nodes (Workers). The scheduling center is responsible for unified management and scheduling decisions; multiple stateless worker nodes form a cluster, responsible for specific task execution, realizing horizontal scaling of processing capacity.

[0243] Distributed task queue: Utilizing Kafka as a distributed message middleware, the scheduling center and worker nodes are decoupled. Task instructions are transmitted through the message queue, and worker nodes automatically consume tasks from the queue, achieving loose coupling and asynchronous processing.

[0244] Failover and elastic scaling: When a new worker node joins the cluster, the load balancer automatically includes it in the scheduling scope, improving system processing capacity. When a worker node fails, the system automatically retrieves the remaining task objects from the pending task queue of the failed node and redistributes them to other healthy nodes.

[0245] Example 2:

[0246] This embodiment is an application of the open-source compliant and efficient scanning method described in Embodiment 1 in the scenario of "user-initiated single package scan".

[0247] I. Implementation Background:

[0248] An operating system development engineer, A, needs to perform an open-source compliance scan on a software package in the "Desktop Operating System V1.0" developed by his team to ensure that it meets open-source compliance requirements and can be safely integrated into the upcoming operating system version.

[0249] II. Implementation Process:

[0250] 1. Task submission.

[0251] 1) Login System: User A logs in to the open source compliance scanning system.

[0252] 2) Create a new scan template: Go to the scan template menu, click the "Create a new scan template" button, enter the scan template name and description, select the required scan plugins, such as open source code snippet reference, dependency component identification, license detection, security vulnerability scanning, etc., and create a new scan template.

[0253] 3) Create a new scan task: Go to the scan task menu, click the "Create new scan task" button, select package scan as the task type, upload the package, select the scan template, and create a new scan task.

[0254] 2. Task assignment.

[0255] After receiving a task request from user A, the system first encapsulates it into a task object. Then, based on a distributed task dispatch algorithm that calculates execution cost, the system selects the optimal task execution node from among many worker nodes and pushes the task object to the queue of tasks to be processed on that node, awaiting processing by the worker node.

[0256] 3. Main task creation.

[0257] The Kafka thread retrieves tasks from the task queue, creates a main scan task, sets the main task status to Pending, and sets the due date to the current time.

[0258] 4. Execution of the main task.

[0259] 1) Preprocessing: The main thread retrieves the due main tasks with a Pending status from the task table, starts the preprocessing process, including decompressing the software package, filtering interfering files, extracting and saving metadata, etc., and transforms the raw input into a standardized scan list.

[0260] 2) Subtask Generation: After preprocessing, the main task first reads the scan template configuration information to obtain a list of scan plugins to be executed. Then, it assembles subtask information for each scan plugin and creates subtasks one by one. The initial status of each subtask is set to Pending, and the expiration time is set to the current time. The parent-child relationship between the subtask and the main task is also saved. There is a one-to-one correspondence between scan plugins and subtasks; the number of subtasks generated corresponds to the number of scan plugins configured in the scan template.

[0261] 5. Subtask execution.

[0262] The main thread periodically retrieves a certain number of due subtasks from the task table. The specific execution logic of these subtasks is determined by the function and params fields in the plugin table, and all subtasks will be executed concurrently.

[0263] 6. Results summary and report generation.

[0264] The inspection thread continuously polls the status of all subtasks under the main task. Once all subtasks have been processed, the inspection thread summarizes the scan results, updates the main task status to "Completed", "Failed", or "Timeout", and generates a scan report.

[0265] Example 3:

[0266] This embodiment is an application of the open-source compliance and efficient scanning method described in Embodiment 1 in the scenario of "PR submission triggering incremental scanning of code repository".

[0267] I. Implementation Background:

[0268] A development team for an open-source operating system uses Gitee as its code hosting platform. After developer B completes the development of a certain functional module and submits a PR to the main branch, an incremental code repository scan needs to be automatically triggered. This scan will only perform open-source compliance scanning on the changed files in the current PR, and the scan results will be fed back to the PR page in real time so that reviewers can quickly determine whether the PR meets open-source compliance requirements and prevent non-compliant code from entering the main branch.

[0269] II. Implementation Process;

[0270] 1. Preliminary preparations.

[0271] 1) Pre-configured incremental scan templates: Users log in to the open-source compliance scanning system and pre-configure the required scan templates so that the Gitee platform can directly specify and use them when calling the open-source compliance scanning API.

[0272] 2) Gitee Integration with Open Source Compliance Scanning System: Users log in to the Gitee platform, go to the WebHooks management page, click the "Add Webhook" button, enter the URL, WebHook password / signing key, select the event, and complete the webhook addition. Simultaneously, in the "Third-Party Integration" module of the open source compliance scanning system, users enter the Gitee access token, repository address, etc., to complete Gitee platform authentication.

[0273] 2. Task triggered.

[0274] When developer B creates a new Pull Request (PR), the Gitee platform automatically sends an HTTP POST request to the open-source compliance scanning system based on pre-added webhook configurations. The request body, in JSON format, contains key information related to the PR, such as the PR number, source branch, and target branch. Upon receiving this request and verifying its legitimacy, the open-source compliance scanning system submits a new scan request.

[0275] 3. Task assignment.

[0276] Upon receiving this task request, the system first encapsulates it into a task object. Then, based on a distributed task dispatch algorithm that calculates execution costs, the system selects the optimal task execution node from among numerous worker nodes and pushes the task object to the queue of tasks to be processed on that node, awaiting processing by the worker node.

[0277] 4. Main task creation.

[0278] The Kafka thread retrieves tasks from the task queue, creates a main scan task, sets the main task status to Pending, and sets the due date to the current time.

[0279] 5. Execution of the main task.

[0280] 1) Preprocessing: The main thread retrieves the due main tasks with the status of Pending from the task table, starts the preprocessing process, including obtaining and downloading changed files, filtering interfering files, extracting and saving metadata, etc., and transforms the raw input into a standardized scan list.

[0281] 2) Subtask Generation: After preprocessing, the main task first reads the scan template configuration information to obtain a list of scan plugins to be executed. Then, it assembles subtask information for each scan plugin and creates subtasks one by one. The initial status of each subtask is set to Pending, and the expiration time is set to the current time. The parent-child relationship between the subtask and the main task is also saved. There is a one-to-one correspondence between scan plugins and subtasks; the number of subtasks generated corresponds to the number of scan plugins configured in the scan template.

[0282] 6. Subtask execution.

[0283] The main thread periodically retrieves a certain number of due subtasks from the task table. The specific execution logic of these subtasks is determined by the function and params fields in the plugin table, and all subtasks will be executed concurrently.

[0284] 7. Summary of results.

[0285] The patrol thread continuously polls the status of all subtasks under the main task. Once all subtasks have been processed, the patrol thread summarizes the scan results and updates the main task status to "Completed", "Failed", or "Timeout".

[0286] 8. Results feedback.

[0287] The system calls the GiteeAPI to send a link to the scan results or scan result information to the PR page, preventing the merging of inappropriate code until the problem is resolved.

[0288] Example 4:

[0289] This embodiment is an application of the open-source, compliant, and efficient scanning method described in Embodiment 1 in the scenario of "full scan of operating system products".

[0290] I. Implementation Background:

[0291] A well-known operating system vendor is about to release its "Desktop Operating System V2.0." This product is massive, containing over 3,000 components and a total codebase exceeding 200GB. To ensure that all components of the operating system comply with open-source compliance requirements, effectively mitigate potential legal risks, and guarantee the product's timely release, the vendor has decided to conduct a full open-source compliance scan of the product. This scan must cover more than ten items, including license identification, dependency component identification, open-source code snippet citation scanning, and security vulnerability scanning, while also meeting the project schedule requirement of "completing the scan and outputting results within 6 hours."

[0292] II. Implementation Process:

[0293] 1. Task submission.

[0294] 1) Login System: User C logs in to the open source compliance scanning system.

[0295] 2) Create a new scan template: Enter the scan template menu, click the "Create a new scan template" button, enter the scan template name and description, check the required scan plugins, such as open source code snippet reference, dependency component identification, license detection, security vulnerability scanning, etc., and create a new scan template "Desktop Operating System V2.0 Full Scan".

[0296] 3) Filtering Interference Files: Enter the interference file filtering menu, add filtering rules, and this scan will first exclude the following specified directories:

[0297] Dependency management directories: bower_components, vendor, jspm_packages, Carthage, Pods, .gradle, deps, _build, .m2, .npm, node_modules;

[0298] Build output directories: build,target,dist,bin,obj,out,debug,.next,.nuxt,.svelte-kit,lib;

[0299] Development environment directories: .pyenv, env, venv, ENV, .venv;

[0300] Cache directories: .cache, .pytest_cache, .nyc_output, .parcel-cache, pycache;

[0301] Document directory: docs;

[0302] IDE configuration directories: .idea, .vs, .vscode-test, .vscode;

[0303] Then, skip the following specified compressed file types: .zip, .rar, .tar, .tar.gz, .tar.7z, .tar.xz, .deb, .rpm, .jar;

[0304] The rules for filtering interfering files are bound to the scanning templates mentioned above.

[0305] 4) Create a new product: Go to the product menu, click the "Create a new product" button, enter the product name, description and version number, select the hosting platform (such as Artifactory, koji, etc.), and create a new product "Desktop Operating System V2.0".

[0306] 5) Create a new product scanning task: Enter the scanning task menu, click the "Create new scanning task" button, select "Product Scan" as the task type, select "Desktop Operating System V2.0" as the product, select "Desktop Operating System V2.0 Full Scan" as the scan template, check "Reuse historical scan results", and create a new scanning task.

[0307] 2. Task assignment.

[0308] 1) Batch creation of software package scanning tasks: After receiving a task request submitted by user C, the system reads the corresponding product configuration information and automatically establishes a connection with the hosting platform. It then retrieves a complete list of software packages for the product (including package name, version, storage path, and other key information) through the platform's open interface. Subsequently, the system iterates through each software package in the list and creates an independent scanning task for each package.

[0309] 2) Batch dispatch of software package scanning tasks: The system encapsulates each software package scanning task into an independent task object in sequence. Based on the distributed task dispatch algorithm based on execution cost calculation, the system dynamically selects the optimal node from multiple worker nodes and pushes the task object to the pending task queue of that node, waiting for the worker node to process it.

[0310] 3. Main task creation.

[0311] The Kafka thread retrieves tasks from the task queue, creates a main scan task, sets the main task status to Pending, and sets the due date to the current time.

[0312] 4. Execution of the main task.

[0313] 1) Preprocessing: The main thread retrieves the due main tasks with a Pending status from the task table, starts the preprocessing process, including downloading and decompressing software packages, filtering interfering files, extracting and saving metadata, etc., and transforms the raw input into a standardized scan list.

[0314] 2) Subtask Generation: After preprocessing, the main task first reads the scan template configuration information to obtain a list of scan plugins to be executed. Then, it assembles subtask information for each scan plugin and creates subtasks one by one. The initial status of each subtask is set to Pending, and the expiration time is set to the current time. The parent-child relationship between the subtask and the main task is also saved. There is a one-to-one correspondence between scan plugins and subtasks; the number of subtasks generated corresponds to the number of scan plugins configured in the scan template.

[0315] 5. Subtask execution.

[0316] The main thread periodically retrieves a certain number of due subtasks from the task table. The specific execution logic of these subtasks is determined by the `function` and `params` fields in the plugin table. Each worker node executes all subtasks concurrently, making full use of computing resources and improving scanning efficiency.

[0317] 6. Results summary.

[0318] The patrol thread continuously polls the status of all subtasks under the main task. Once all subtasks have been processed, the patrol thread summarizes the scan results and updates the main task status to "Completed", "Failed", or "Timeout".

[0319] 7. Flexible expansion and contraction.

[0320] As package scanning tasks are continuously created, dispatched, and executed, Prometheus detects that the task queue length and CPU / memory usage of all current worker nodes have exceeded the preset warning values, and the average task processing time is also showing an upward trend. The system automatically increases the number of worker node replicas (2->4->8) to cope with the large number of scanning tasks.

[0321] As the package scanning task gradually comes to an end, when the task queue length of all worker nodes, CPU / memory utilization are all below the preset safety value, and the average task processing time shows a downward trend, the system automatically reduces the number of worker node replicas (8->4->2) to save hardware resources.

[0322] 8. High reliability and fault tolerance.

[0323] 1) Reassignment of tasks at abnormal nodes:

[0324] During the scan, worker node Worker-03 crashed due to a hardware failure.

[0325] The scheduling center monitors the node status in real time through the Kubernetes API. If Worker-03 is unresponsive for more than 1 minute, it is marked as an "abnormal node" and the recovery process is triggered.

[0326] The system automatically retrieves the remaining task objects from the pending task queue of Worker-03 and redistributes them to other healthy nodes.

[0327] 2) Automatic retry of failed tasks:

[0328] A certain "open source code fragment reference scanning" subtask failed to execute due to the temporary unavailability of the knowledge base, triggering an automatic retry mechanism: the first retry interval is 2 minutes, the second retry interval is 4 minutes, the third retry interval is 8 minutes, and the knowledge base is restored during the third retry, and the subtask is executed successfully.

[0329] 9. Knowledge base interaction optimization.

[0330] To improve knowledge base access efficiency, the system employs a local caching strategy, caching frequently used knowledge base data (such as Top 20 licenses, Top 100 components, Top 100 security vulnerabilities, etc.). Network requests to update data are only initiated when the cache expires. Simultaneously, the system uses a batch query mechanism to optimize the interaction process with the knowledge base, reducing access frequency and response time, and improving overall scanning performance.

[0331] 10. Summary of results.

[0332] At 4 hours and 50 minutes into the task, the system detected that all package scanning tasks had been completed, automatically started the results aggregation process, and generated a standardized compliance report.

[0333] Example 5:

[0334] This embodiment proposes an open-source, compliant, and efficient scanning system for operating systems, including:

[0335] Module splitting and combination: Open source compliance scanning is broken down into several scanning items, each of which is implemented by an independent scanning plugin; different scanning plugins can be combined through scanning templates to achieve configurable differentiated scanning;

[0336] Trigger Scan Module: After triggering a scan task, determine the main task type, obtain the scan object based on the main task type, and read the scan template corresponding to the main task type;

[0337] Reuse Judgment Module: Extracts metadata information of scanned objects, generates Uniform Resource Locator (PURL) as the identifier of scanned objects, and uses the PURL to determine whether to execute the historical scan result reuse mechanism. If so, the scanning task is stopped and the corresponding historical scan result is returned directly; otherwise, it enters the distributed task module.

[0338] Distributed Task Module: Employs a distributed task scheduling framework. Based on the scanning plugins configured in the scanning template corresponding to the main task type, it creates subtasks for the scanning items corresponding to the scanning plugins, performs distributed subtask dispatch based on execution cost calculation, and manages the scheduling and execution of subtasks. It summarizes and displays the scanning results and stores the scanning tasks and results as historical scanning data of successful execution.

[0339] In the splitting and combining module, the relationship between scanning templates and scanning plugins is maintained through three tables: a template table that stores basic information about scanning templates, a plugin table that stores basic information about scanning plugins, and a template-plugin relationship table that stores the relationship between scanning templates and scanning plugins.

[0340] The trigger scan task in the trigger scan module includes an active initiation mode and a passive trigger mode. The main task types of the active initiation mode include package scanning, code repository scanning, and operating system scanning. The main task types of the passive trigger mode include pipeline triggering, artifact warehousing triggering, and PR / MR triggering.

[0341] In the triggered scan module, after obtaining the scan object based on the main task type, interference file filtering is performed. The filtering methods include: excluding specified directories, excluding specified file types, scanning specified file types, and skipping specified compressed file types.

[0342] The Uniform Resource Locator (PURL) in the reuse judgment module includes:

[0343] Depending on the package type of the software package or code repository, extract metadata information of the scanned object from the debian / control, debian / changelog, and debian / copyright files in the Debian package, or the .spec file in the RPM package, and the code repository URL string; generate a PURL that conforms to the specification based on the metadata information.

[0344] The mechanism for determining whether to reuse historical scan results in the reuse judgment module includes:

[0345] Based on the identifier PURL of the scanned object and the ID of the scan template corresponding to the main task type, query the successfully executed historical scan data to see if there are any historical scan tasks for the same scanned object and their corresponding historical scan results. If so, further determine whether the scan plugin in the scan template corresponding to the main task type is completely consistent with the plugin used in the historical scan task. If they are completely consistent, return the historical scan results directly.

[0346] The distributed task module describes the use of a distributed task scheduling framework, including:

[0347] Establish cloud-native deployment, perform distributed task scheduling, create a three-thread collaboration mechanism of Kafka thread, main thread, and patrol thread, and achieve fault tolerance through fine-grained management of task status.

[0348] Cloud-native deployments include:

[0349] Standardized container packaging based on Docker;

[0350] Based on Kubernetes orchestration capabilities, differentiated deployment strategies are designed for the characteristics of different components, including a scheduling center, worker nodes, data services, and message middleware.

[0351] Introducing the Prometheus monitoring system, collecting dual-dimensional metrics of task load and node resources, and building custom elastic scaling rules;

[0352] By adopting GitOps continuous deployment, all Kubernetes resource inventories are stored in a Git repository, enabling versioned configuration management, automatic synchronization, and environment isolation.

[0353] Distributed task scheduling includes:

[0354] Construct a two-level task model that includes main tasks and sub-tasks;

[0355] The distributed task dispatch based on execution cost is calculated using the following formula: Where C represents the execution cost of the scanning task at a certain working node; S represents the size of the scanned object; n represents the number of scanned items, and i represents the i-th scanned item; The time consumption weight of the i-th scan item is pre-set based on the computational complexity and resource consumption of the scan item; P represents the performance index of the worker node, calculated based on the number of CPU cores, memory size, and throughput.

[0356] Each worker node creates a Kafka thread, a main thread, and a patrol thread upon startup, and completes the scanning task through a three-thread collaborative mechanism.

[0357] The open-source compliance and high-efficiency scanning system for operating systems proposed in this embodiment can execute the open-source compliance and high-efficiency scanning method for operating systems described in Embodiment 1, and has the same technical effects as Embodiment 1.

[0358] The above-described embodiments are merely preferred embodiments of the present invention and are only used to help understand the method and core ideas of this application. The scope of protection of the present invention is not limited to the above embodiments, and all technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. An open-source, compliant, and efficient scanning method for operating systems, characterized in that, include: S1. Break down open-source compliance scanning into several scanning items, each with its own independent scanning plugin; combine different scanning plugins using scanning templates to achieve configurable, differentiated scanning. S2. After triggering the scanning task, determine the main task type, obtain the scanning object according to the main task type, and read the scanning template corresponding to the main task type; S3. Extract the metadata information of the scanned object and generate a Uniform Resource Locator (PURL) as the identifier of the scanned object. Use the PURL to determine whether to execute the historical scan result reuse mechanism. If yes, stop the scanning task and directly return the corresponding historical scan result. Otherwise, proceed to step S4. The method for generating a Uniform Resource Locator (PURL) includes: extracting metadata information of the scanned object from the debian / control, debian / changelog, and debian / copyright files in a Debian package, or the .spec file in an RPM package, and the URL string of the code repository, based on the different package types of the software package or code repository; and generating a PURL that conforms to the specification based on the metadata information. S4. A distributed task scheduling framework is adopted. Based on the scanning plugins configured in the scanning template corresponding to the main task type, subtasks are created for the scanning items corresponding to the scanning plugins. Distributed subtask dispatch is performed based on execution cost calculation, and the scheduling and execution of subtasks are managed. Scan results are summarized and displayed, and scanning tasks and results are stored as historical scan data of successful execution. The method of adopting the distributed task scheduling framework includes: establishing a cloud-native deployment, performing distributed task scheduling, creating a three-thread collaboration mechanism of Kafka thread, main thread, and patrol thread, and achieving fault tolerance through fine-grained management of task status. The distributed task scheduling includes: constructing a two-level task model including main tasks and subtasks; and performing distributed task dispatch based on execution cost calculation, the calculation formula being: Where C represents the execution cost of the scanning task at a certain working node; S represents the size of the scanned object; n represents the number of scanned items, and i represents the i-th scanned item; The time consumption weight of the i-th scan item is pre-set based on the computational complexity and resource consumption of the scan item; P represents the performance index of the worker node, calculated based on the number of CPU cores, memory size, and throughput; each worker node creates a Kafka thread, a main thread, and a patrol thread at startup, and completes the scan task through a three-thread collaborative mechanism.

2. The open-source, compliant, and efficient scanning method for operating systems according to claim 1, characterized in that, In step S1, the relationship between the scanning template and the scanning plugin is maintained through three tables: a template table that stores basic information about the scanning template, a plugin table that stores basic information about the scanning plugin, and a template-plugin relationship table that stores the relationship between the scanning template and the scanning plugin.

3. The open-source, compliant, and efficient scanning method for operating systems according to claim 1, characterized in that, The triggering of the scanning task in step S2 includes an active initiation mode and a passive triggering mode. The main task types of the active initiation mode include package scanning, code repository scanning, and operating system scanning. The main task types of the passive triggering mode include production line triggering, product warehousing triggering, and PR / MR triggering.

4. The open-source, compliant, and efficient scanning method for operating systems according to claim 1, characterized in that, In step S2, after obtaining the scan object according to the main task type, interference file filtering is performed. The filtering methods include: excluding specified directories, excluding specified file types, scanning specified file types, and skipping specified compressed file types.

5. The open-source, compliant, and efficient scanning method for operating systems according to claim 1, characterized in that, The methods for determining whether to execute the historical scan result reuse mechanism in step S3 include: Based on the identifier PURL of the scanned object and the ID of the scan template corresponding to the main task type, query the successfully executed historical scan data to see if there are any historical scan tasks for the same scanned object and their corresponding historical scan results. If so, further determine whether the scan plugin in the scan template corresponding to the main task type is completely consistent with the plugin used in the historical scan task. If they are completely consistent, return the historical scan results directly.

6. The open-source compliance and high-efficiency scanning method for operating systems according to claim 1, characterized in that, Methods for establishing cloud-native deployments include: Standardized container packaging based on Docker; Based on Kubernetes orchestration capabilities, differentiated deployment strategies are designed for the characteristics of different components, including a scheduling center, worker nodes, data services, and message middleware. Introducing the Prometheus monitoring system, collecting dual-dimensional metrics of task load and node resources, and building custom elastic scaling rules; By adopting GitOps continuous deployment, all Kubernetes resource inventories are stored in a Git repository, enabling versioned configuration management, automatic synchronization, and environment isolation.

7. An open-source, compliant, and efficient scanning system for operating systems, characterized in that: include: Module splitting and combination: Open source compliance scanning is broken down into several scanning items, each of which is implemented by an independent scanning plugin; different scanning plugins can be combined through scanning templates to achieve configurable differentiated scanning; Trigger Scan Module: After triggering a scan task, determine the main task type, obtain the scan object based on the main task type, and read the scan template corresponding to the main task type; Reuse Judgment Module: Extracts metadata information of scanned objects, generates Uniform Resource Locator (PURL) as the identifier of scanned objects, and uses the PURL to determine whether to execute the historical scan result reuse mechanism. If so, the scanning task is stopped and the corresponding historical scan result is returned directly; otherwise, it enters the distributed task module. The method for generating a Uniform Resource Locator (PURL) includes: extracting metadata information of the scanned object from the debian / control, debian / changelog, and debian / copyright files in a Debian package, or the .spec file in an RPM package, and the URL string of the code repository, based on the different package types of the software package or code repository; and generating a PURL that conforms to the specification based on the metadata information. Distributed Task Module: Employing a distributed task scheduling framework, this module creates subtasks for the corresponding scan items based on the scan plugins configured in the scan template corresponding to the main task type. It then performs distributed subtask dispatch based on execution cost calculation, managing the scheduling and execution of subtasks. Scan results are summarized and displayed, and scan tasks and results are stored as historical scan data of successful execution. The distributed task scheduling framework includes: establishing a cloud-native deployment, performing distributed task scheduling, creating a three-thread collaboration mechanism (Kafka thread, main thread, and patrol thread), and achieving fault tolerance through fine-grained task status management. The distributed task scheduling includes: constructing a two-level task model including main tasks and subtasks; and performing distributed task dispatch based on execution cost calculation, using the following formula: Where C represents the execution cost of the scanning task at a certain working node; S represents the size of the scanned object; n represents the number of scanned items, and i represents the i-th scanned item; The time consumption weight of the i-th scan item is pre-set based on the computational complexity and resource consumption of the scan item; P represents the performance index of the worker node, calculated based on the number of CPU cores, memory size, and throughput; each worker node creates a Kafka thread, a main thread, and a patrol thread at startup, and completes the scan task through a three-thread collaborative mechanism.

Images