Method and device for collecting and processing component metadata

Abstract

The invention discloses a method and a device for collecting and processing component metadata, and relates to the technical field of computers. A specific embodiment of the method comprises the following steps: determining a package source of each component in a component library, and acquiring component package data from each component package source or an index file of each component package source according to a specific acquisition tool; sending each component packet data to the message queue cluster through a data consumption pipeline provided by the message queue cluster; according to a specific consumption tool, each piece of component package data is obtained from the message queue cluster through the data consumption pipeline, information of specific attributes is obtained from each piece of component package data, and component metadata is obtained. According to the embodiment, the component metadata acquisition process is standardized and simplified, multiple languages are supported, the development process is accelerated, and a solid foundation is laid for constructing a high-quality, safe and maintainable software system.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for collecting and processing component metadata. Background Art

[0002] Component metadata is essential for software development and maintenance, as it provides detailed information about components. It is very necessary to collect, organize, and provide detailed information about various software projects through component metadata, improve the efficiency and transparency of open source software management, help users better understand software, comply with license compliance requirements, and reduce security vulnerability risks. However, existing component metadata collection methods are usually limited to a single component type or implemented through simple crawlers, and lack general and scalable capabilities. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a method and device for collecting and processing component metadata, which can at least solve the problem that the method for collecting component metadata in the prior art is not universal and scalable.

[0004] To achieve the above-mentioned purpose, according to one aspect of an embodiment of the present invention, a method for collecting and processing component metadata is provided, including: determining the package source of each component in the component library, and obtaining component package data from each component package source or an index file of each component package source based on a specific collection tool; sending each component package data to the message queue cluster through a data consumption pipeline provided by the message queue cluster; obtaining each component package data from the message queue cluster through the data consumption pipeline based on a specific consumption tool, obtaining specific attribute information from each component package data, and obtaining component metadata.

[0005] Optionally, the data consumption pipeline provided by the message queue cluster sends each component package data to the message queue cluster, including: determining the component type according to the component package name in each component package data; sending each component package data to a target topic corresponding to the component type in the message queue cluster through the data consumption pipeline provided by the message queue cluster according to the correspondence between the preset component type and the topic; obtaining each component package data from the message queue cluster through the data consumption pipeline includes: obtaining each component package data from the target topic through the data consumption pipeline.

[0006] Optionally, the method also includes: generating a component package name list based on the component package names, and reading component package data in batches according to the component package names in the component package name list; sending the read component package data in batches to the topic of the message queue cluster through a data consumption pipeline provided by the message queue cluster; and updating the component package name reading position in the component package name list.

[0007] Optionally, the specific consumption tool includes one or more of a component label collection service, a component dependency collection service, a component license collection service, a component vulnerability service, and a component warehouse service; the obtaining of information of specific attributes from each component package data to obtain component metadata includes: determining the language type according to the component package name in each component package data according to the component label collection service, and marking each component package data with a label pre-configured for the language type to obtain component label information; determining the component type according to the component package name according to the component dependency collection service, and then determining the dependency relationship between component packages according to the component package name or component type, and constructing a component dependency tree according to the dependency relationship to obtain component dependency information; obtaining component license information from the component package source of each component package data according to the component license collection service; obtaining component access links from the component package data according to the component vulnerability service, displaying a page corresponding to the component access links, obtaining security vulnerability information of the component packages, and obtaining security vulnerability information of other component packages that have dependencies with the component packages; According to the component repository service, relevant information of the source code of the component package source is obtained from the repository; component metadata is obtained based on one or more of the component package name, component tag information, component dependency information, component license information, component security vulnerability information, and component source code related information.

[0008] Optionally, after obtaining the component metadata, the method further includes: storing each component metadata in a distributed database, and storing each component package data in an object storage service.

[0009] Optionally, the component package data and component metadata also include a version number. After obtaining the component metadata, the method also includes: in the process of generating a software bill of materials document, detecting whether each field of each collected component metadata has a missing field value; for target component metadata with missing field values, obtaining the target component package name and target version number from the target component metadata; calling the component metadata interface service to determine the component metadata set corresponding to the target component package name from a distributed database; determining the historical version component metadata in the component metadata set that is before the target version number, obtaining the field value corresponding to each field from the historical version component metadata, and supplementing the field value to the missing field value position of the software bill of materials document.

[0010] Optionally, after obtaining the component metadata, the method also includes: determining the score of each attribute based on the missing field value under each attribute in the component metadata; obtaining the weight value pre-configured for each attribute, and calculating the comprehensive score of each component by weighted summation; receiving input component filtering conditions, obtaining all components that meet the component filtering conditions, and sorting and displaying them in descending order according to the component's comprehensive score; and removing components whose comprehensive scores are less than a preset threshold.

[0011] To achieve the above-mentioned purpose, according to another aspect of an embodiment of the present invention, a component metadata collection and processing device is provided, including: an acquisition module, used to determine the package source of each component in the component library, and obtain component package data from each component package source or the index file of each component package source according to a specific acquisition tool; a transmission module, used to send each component package data to the message queue cluster through a data consumption pipeline provided by the message queue cluster; a consumption module, used to obtain each component package data from the message queue cluster through the data consumption pipeline according to a specific consumption tool, obtain specific attribute information from each component package data, and obtain component metadata.

[0012] Optionally, the transmission module is used to: determine the component type based on the component package name in each component package data; according to the correspondence between the preset component type and the topic, send each component package data to the target topic corresponding to the component type in the message queue cluster through the data consumption pipeline provided by the message queue cluster; the consumption module is used to: obtain each component package data from the target topic through the data consumption pipeline.

[0013] Optionally, the transmission module is also used to: generate a component package name list based on the component package name, and read the component package data in batches according to the component package names in the component package name list; send the read component package data in batches to the topic of the message queue cluster through the data consumption pipeline provided by the message queue cluster; and update the component package name reading position in the component package name list.

[0014] Optionally, the specific consumption tool includes one or more of a component label collection service, a component dependency collection service, a component license collection service, a component vulnerability service, and a component warehouse service; the consumption module is used to: determine the language type according to the component package name in each component package data according to the component label collection service, and use the label pre-configured for the language type to mark each component package data to obtain component label information; determine the component type according to the component package name according to the component dependency collection service, and then determine the dependency relationship between component packages according to the component package name or component type, build a component dependency tree according to the dependency relationship, and obtain component dependency information; obtain component license information from the component package source of each component package data according to the component license collection service; obtain component access links from the component package data according to the component vulnerability service, display the page corresponding to the component access links, obtain security vulnerability information of the component packages, and obtain security vulnerability information of other component packages that have dependencies with the component packages; obtain relevant information of the source code of the component package source from the warehouse according to the component warehouse service; obtain component metadata based on one or more of the component package name, component label information, component dependency information, component license information, component security vulnerability information, and component source code related information.

[0015] Optionally, the device also includes a storage module, which is used to: store each component metadata in a distributed database, and store each component package data in an object storage service.

[0016] Optionally, the component package data and component metadata also include a version number, and the device also includes a document module, which is used to: in the process of generating a software bill of materials document, detect whether there is a missing field value in each field of each collected component metadata; for target component metadata with missing field values, obtain the target component package name and target version number from the target component metadata; call the component metadata interface service to determine the component metadata set corresponding to the target component package name from the distributed database; determine the historical version component metadata in the component metadata set that is before the target version number, obtain the field value corresponding to each field from the historical version component metadata, and supplement the field value to the missing field value position of the software bill of materials document.

[0017] Optionally, the device also includes a component recommendation module, which is used to: determine the score of each attribute based on the missing field value under each attribute in the component metadata; obtain the weight value pre-configured for each attribute, and calculate the comprehensive score of each component by weighted summation; receive input component filtering conditions, obtain all components that meet the component filtering conditions, and sort and display them in order from large to small according to the component's comprehensive score; and remove components whose comprehensive scores are less than a preset threshold.

[0018] To achieve the above objective, according to another aspect of an embodiment of the present invention, an electronic device for collecting and processing component metadata is provided.

[0019] The electronic device of an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement any of the component metadata collection and processing methods described above.

[0020] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, any of the above-mentioned component metadata collection and processing methods is implemented.

[0021] To achieve the above object, according to another aspect of an embodiment of the present invention, a computer program product is provided. A computer program product of an embodiment of the present invention includes a computer program, and when the program is executed by a processor, the component metadata collection and processing method provided by the embodiment of the present invention is implemented.

[0022] According to the solution provided by the present invention, an embodiment of the above invention has the following advantages or beneficial effects: solving the problem of the lack of a unified component metadata collection framework, and setting multiple collection attributes according to needs, so as to standardize and simplify the component metadata collection process, support multiple languages ​​as a whole, accelerate the development process, and lay a solid foundation for building high-quality, secure and maintainable software systems.

[0023] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0025] Figure 1 It is a main flow chart of a component metadata collection and processing method according to an embodiment of the present invention;

[0026] Figure 2 is a flow chart of an optional component metadata collection and processing method according to an embodiment of the present invention;

[0027] Figure 3 is a flowchart of another optional component metadata collection and processing method according to an embodiment of the present invention;

[0028] Figure 4 is a schematic diagram of a collection framework of a specific component metadata according to an embodiment of the present invention;

[0029] Figure 5 This is a schematic diagram of main modules of a device for collecting and processing component metadata according to an embodiment of the present invention;

[0030] Figure 6 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0031] Figure 7 It is a schematic diagram of the structure of a computer system of a mobile device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0033] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0034] See also Figure 1 , which shows a main flow chart of a component metadata collection and processing method provided by an embodiment of the present invention, including the following steps:

[0035] S101: Determine the package source of each component in the component library, and obtain component package data from each component package source or an index file of each component package source according to a specific acquisition tool;

[0036] S102: Send each component package data to the message queue cluster through the data consumption pipeline provided by the message queue cluster;

[0037] S103: According to a specific consumption tool, each component package data is obtained from the message queue cluster through the data consumption pipeline, and specific attribute information is obtained from each component package data to obtain component metadata.

[0038] In the above implementation, a component refers to a relatively independent unit in software development, which has specific functions and interfaces and can be reused in different programs or systems. The purpose of component design is to promote modularity and reusability and simplify the software development and maintenance process. A component can be a small part of the software code or a complete program, but is usually designed as an independent unit that performs a specific task.

[0039] Component metadata is descriptive data about a component, which is usually used to define, describe, and manage the properties and behaviors of a component. It provides detailed information about the component, including but not limited to version numbers, dependencies, license information, author and maintainer information, security vulnerability reports, update history, etc. It is very important for developers, project managers, and compliance auditors. Component metadata not only helps understand and evaluate the applicability of components, but is also critical to ensuring the security, compliance, and ongoing maintenance of software projects. Component information and component metadata can be used interchangeably in some cases, but generally component information is a broader concept that covers all component-related details, while component metadata is a subset that is more focused on describing and managing component properties and behaviors.

[0040] For step S101, the present solution is first provided with a component library, including components developed by the enterprise itself and open source components outside the enterprise, so the component library usually includes multiple components, such as maven, npm, golang, pypi, etc. The present solution is also provided with a specific collection tool, such as a component package collector, which is responsible for obtaining full and incremental component package data from various types of component package sources or component package source index files (similar to directories) such as maven, npm, golang, pypi, etc., mainly including component package names and component package version numbers. Taking maven as an example, if you want to obtain all package information of the globally open source maven, you must first know the package name and version number, but the package name and version number are obtained from the full and incremental index files of the central warehouse maven. Among them, the package source refers to an online warehouse or location for storing and distributing software packages. These software packages usually contain precompiled code, libraries, frameworks, or all dependencies of an application, which is convenient for developers to install and update. Different programming languages ​​and ecosystems have their own official or third-party maintained package sources.

[0041] For step S102, the component package data is sent to a message queue cluster, such as Kafka. Kafka is a distributed stream processing platform that is often used to build real-time data pipelines and stream applications, and can process large amounts of real-time data streams. Sending this data to the Kafka cluster means converting the data into messages and publishing them to specific topics so that subsequent consumer services can subscribe to and process these messages. Therefore, it is necessary to first determine the component type. In this solution, the component type is determined based on the component package name, such as based on the suffix of the component package name. In software development, different file suffixes represent different types of components or resources.

[0042] For example, if the suffix is ​​.js, it means the component type is a JavaScript source code file; if the suffix is ​​.jsx, it means the component type is a React component; if the suffix is ​​.ts, it means the component type is a TypeScript source code file; if the suffix is ​​.vue, it means the component type is a Vue.js single-file component. Each suffix corresponds to a specific component type, which helps developers organize and understand the code structure when working on large projects. For example, in a project using React, when you see a .jsx file, you can be sure that this is a file containing a React component without having to look further into the file contents. Similarly, a .vue file in a Vue.js project represents a Vue component, which contains templates, logic, and styles. In this way, the suffix not only helps to process files correctly, but also facilitates collaboration and code reading among team members.

[0043] Kafka pre-sets a corresponding topic for each component type. For example, JavaScript source code files and React components correspond to Topic 1, TypeScript source code files correspond to Topic 2, and Vue.js single-file components correspond to Topic 3. This allows component package data of different component types to be quickly sent to the corresponding topics of Kafka for independent storage. Subsequent component package data is also obtained from different topics.

[0044] In actual operation, there are many components, and the number of component package data obtained is also multiple. In order to improve efficiency and reduce the consumption of system resources, the component package data is not sent to the Topic of Kafka one by one, but is read in batches in batches, so as to reduce the number of reads and improve the overall processing speed. This solution sets a component package name list, so that the component package data will be read in batches according to the component package name list. After the batch-read component package data is serialized (converted into a byte stream), it is sent to the specified Topic of the Kafka cluster through the data consumption pipeline provided by Kafka. This step ensures asynchronous processing and decoupling of the data. After the sending is completed, the component package name list is updated to record the last read position, ensuring that even if the service is interrupted for some reason, it can continue from the last stop and only process new data, thereby avoiding the situation of reading duplicate data after the service is terminated, and ensuring the consistency and integrity of the data.

[0045] For step S103, this solution aims to collect metadata from different types of components, but these components may be implemented using different programming languages ​​and paradigms. Therefore, it is necessary to support multiple programming paradigms or different collection methods to realize the collection and processing of component metadata. This solution sets up specific consumption tools, such as component package consumption service groups, which are responsible for consuming messages from Kafka and calling component label collection services, component dependency collection services, component license collection services, component vulnerability services, and component warehouse services to obtain component label information, component dependency information, component warehouse information, component license information, and component vulnerability information. This method takes advantage of different paradigms and provides a more flexible and powerful solution, which can actually be expanded according to specific needs, such as adding other services.

[0046] 1. Component label collection service is responsible for adding labels to component package data of each different language type. These labels can be used to describe the characteristics and uses of the component. In this solution, the language type is preferably determined by the suffix of the component package name. Taking the suffix .jar or .war as an example, Java component packages are usually packaged into JAR (Java Archive) files or WAR (Web Application Archive) files. A .jar file is a ZIP compressed file containing compiled .class files and other resources, while a .war file is a ZIP compressed file containing all components of a Web application, including .jsp pages, .class files, static resources, and configuration files. In addition, it also provides the function of collecting and creating labels, such as pre-establishing labels corresponding to each language type, and providing the function of updating and deleting labels.

[0047] In addition, it also supports classification management of tags, such as functional tags, technical tags, version tags, etc. 1) Functional tags: usually used to describe the specific functions provided by software or components. For example, a component may have functional tags such as "database connection", "user authentication" or "payment processing". These tags can help developers and project managers quickly identify and locate components with specific functions, making it easier to choose appropriate tools or modules in the project. 2) Technical tags: used to identify the technology stack or programming language used by the component or project. For example, "Java", "Spring Boot", "React" or "Node.js" can be used as technical tags. These tags are crucial to understanding the implementation details, dependencies and compatibility of components, and are also convenient for finding developers familiar with related technologies. 3) Version tags: used to identify the specific version of the software or component. This usually includes the major version number, minor version number and revision number, such as "v1.2.3". Version tags are very important for tracking change history, managing dependencies and ensuring the stability and security of software. In a multi-team collaborative environment, clear version tags can avoid problems caused by version conflicts.

[0048] 2. Component dependency collection service, responsible for collecting, managing and parsing the dependencies of component packages of various language types, including direct dependencies and transitive dependencies. In order to efficiently and accurately collect dependency information of open source components, two implementation strategies are designed to automatically identify and collect component dependencies based on different component types and package suffixes, and finally build a detailed dependency topology diagram.

[0049] Embodiment 1: Collect dependency information by component type. Collect the component information of its dependencies according to the type of component. Each type of component usually has a specific dependency management tool. First, obtain the component type, for example: Maven library, Python package, Node.js module, etc. Secondly, select a suitable dependency management tool according to the component type, for example, Maven tool for Java components, pip tool for Python packages. Then use the selected dependency management tool to go deep into the component to collect the component's dependency package information, including dependency package name, version, dependency package parent dependency, dependency package child dependency information, etc. Finally, the complete component dependency data is collected, and a dependency topology diagram is formed through the parent-child relationship.

[0050] Embodiment 2: Collect dependency information by matching the suffix of the component package name. Identify and collect the dependency relationship through the suffix of the component package name. Common file suffixes include .jar .whl, etc. First, identify the language type through the component file suffix, such as .jar for java and .whl for python. Secondly, identify the key configuration files in the component package of known language types. For example, the java component package needs to identify the pom.xml file in the package, and the python component package needs to identify the METADATA file in the package. These files record the core dependency information of the component. Finally, from the located configuration file, accurately extract the component dependency data, including direct dependencies and indirect dependencies, and finally organize it into a clear dependency relationship to complete the collection of component dependency information.

[0051] Through the implementation of the above two strategies, not only can the dependency information collection needs of different types of components be fully covered, but also the automation and intelligence of the collection process can be ensured. The final dependency topology diagram will provide solid data support for project construction, maintenance and upgrade. In addition, a dependency management service is provided to provide the function of adding and updating dependencies.

[0052] 3. Component license collection service is responsible for collecting and managing the license information of components of various language types, and ensuring that the use of components complies with the requirements of open source licenses based on the component license information. License management is used to provide the functions of adding, updating, and deleting license information. License parsing includes collecting and parsing component license information and identifying license types.

[0053] The license information of a component refers to the legal documents that come with a software component (usually a library, framework or other reusable software module), which defines the conditions for the use, modification and distribution of the component, and usually includes the following points: Copyright information: declares who owns the copyright of the software. Usage permissions: states whether the user has the right to use the component and under what conditions. Modification permissions: whether the user is allowed to modify the source code of the component. Distribution permissions: whether the user has the right to distribute the component or its modified version to others. Limitation of liability: usually states that the software is provided "as is" without any express or implied warranty. Compatibility: some licenses may require derivative works to also use the same license, or a license that is compatible with it.

[0054] 4. Component vulnerability service is responsible for collecting, managing and detecting security vulnerability information of components and their direct and indirect dependent components, and providing vulnerability repair suggestions and security reports. Vulnerability detection is used to detect known vulnerabilities in components and provide vulnerability lists and detailed information. Taking the dependency relationship between components AB as an example, obtain A's URL (Uniform Resource Locator) from A's component package data, access the URL, enter A's official website page, and obtain A's security vulnerability information from the official website page. For B, the dependent component dependency tree is determined, and security vulnerability information usually needs to be obtained from B's official website. This solution also has a vulnerability management function to provide the addition, update, and deletion of vulnerability information. A vulnerability repair suggestion function is also set up to provide vulnerability repair suggestions and patch information to help repair vulnerabilities, such as pre-configuring different vulnerability repair strategies for different security vulnerability types. After that, a special scheduled data patching service is called to repair vulnerabilities through vulnerability repair strategies.

[0055] 5. Component repository service, responsible for collecting relevant information of the source code of the component package source from the git repository, including the source code repository address, author, number of issues, activity and other related information.

[0056] All the above-mentioned metadata collection services are performed through the consumer service group, and then the component metadata is obtained based on one or more of the component package name, component tag information, component dependency information, component license information, component security vulnerability information, and component source code related information, preferably all of them. The component metadata is stored in a distributed database, and the component package data is stored in the object storage service OSS (full name Object Storage Service).

[0057] In this solution, components in the component library can also be scanned regularly. For the same component package name, the version number is analyzed to see if there is an update. If an update is detected, the above operations of obtaining component package data and component metadata are performed to obtain a new component metadata and component package data for storage. Therefore, whether it is a distributed database or OSS, there is information under multiple different version numbers for the same component package name.

[0058] In addition, for the stored data, some fields may be empty, that is, there are missing field values. The program scans and finds that it is incomplete and needs to be updated. Specifically, it triggers data re-crawling to regularly update the code repository data. If the update service is successfully executed for 3 consecutive times (just an example, the actual value is adjustable) but the data is still missing, the update will no longer be triggered.

[0059] The method provided in the above embodiment sets up a unified component metadata collection framework, and sets multiple collection attributes according to needs, including but not limited to component tag information, dependency information, source code related information, security vulnerability information, and license information, and supports attribute extension, so as to standardize and simplify the component metadata collection process, support multiple languages ​​as a whole, accelerate the development process, and lay a solid foundation for building high-quality, secure, and maintainable software systems.

[0060] See also Figure 2 , shows a schematic flow chart of an optional component metadata collection and processing method according to an embodiment of the present invention, comprising the following steps:

[0061] S201: Determine the package source of each component in the component library, and obtain component package data from each component package source or an index file of each component package source according to a specific acquisition tool;

[0062] S202: Send each component package data to the message queue cluster through the data consumption pipeline provided by the message queue cluster;

[0063] S203: According to a specific consumption tool, obtain each component package data from the message queue cluster through the data consumption pipeline, obtain specific attribute information from each component package data, and obtain component metadata;

[0064] S204: In the process of generating the software bill of materials document, detecting whether each field of each component metadata collected has a missing field value;

[0065] S205: For target component metadata with missing field values, obtain the target component package name and target version number from the target component metadata;

[0066] S206: calling a component metadata interface service to determine a component metadata set corresponding to the target component package name from a distributed database;

[0067] S207: Determine the historical version component metadata that is before the target version number in the component metadata set, obtain the field value corresponding to each field from the historical version component metadata, and supplement the field value to the missing field value position of the software bill of materials document.

[0068] In the above embodiment, for steps S201 to S203, see Figure 1 The description shown is not repeated here.

[0069] SBOM (Software Bill of Materials) refers to a list of software components that are recorded and tracked in software development and supply chain management. SBOM aims to provide detailed information about software components, such as name, version, vendor, license information, and dependencies. It is similar to the bill of materials in manufacturing, which is used to describe the components of a product. SBOM is of great significance in software development and maintenance, especially in terms of security, compliance, and risk management.

[0070] Ordinary metadata collection only collects metadata for display, and its application value is not high. In addition, the existing component metadata collection solution cannot be directly connected or integrated with the actual use scenario of SBOM document generation, so it is impossible to improve the richness and accuracy of SBOM document content through component metadata. This solution sets up a solution to this problem, which can integrate component metadata with the SBOM system and supplement a series of indicators such as component information, dependent component information, dependent component security vulnerabilities, and dependent component licenses in the SBOM system.

[0071] For steps S204 to S207, this solution sets up an SBOM document generation service, which is used to query the component metadata API (Application Programming Interface) service to complete the component data when the collected open source component metadata field content is missing during SBOM document generation. Among them, the component metadata API service is used to provide external capabilities such as component query, component addition, deletion, and modification, and component online and offline update through the API interface.

[0072] For example, if the license data of an open source component C in the SBOM document is empty, you can query the component metadata API service to obtain the component metadata set with the same name from the distributed database according to the component package name C of the open source component. Assuming the version number is 1.2.3, obtain the license data from the component metadata of historical versions such as 1.2.2 and 1.2.1 of the component metadata set, and then add the license data to the license data of version 1.2.3 in the SBOM document to increase the integrity of the SBOM document.

[0073] The method provided in the above embodiment integrates SBOM documents and component metadata collection to improve the information integrity, consistency, automation efficiency, security, compliance and risk management capabilities of SBOM documents. This integration helps to manage software components more comprehensively and effectively, enhance the transparency and traceability of the software supply chain, and thus improve the overall software quality and security.

[0074] See also Figure 3, shows a schematic flow chart of another optional component metadata collection and processing method according to an embodiment of the present invention, comprising the following steps:

[0075] S301: Determine the package source of each component in the component library, and obtain component package data from each component package source or an index file of each component package source according to a specific acquisition tool;

[0076] S302: Send each component package data to the message queue cluster through the data consumption pipeline provided by the message queue cluster;

[0077] S303: According to a specific consumption tool, obtain each component package data from the message queue cluster through the data consumption pipeline, obtain specific attribute information from each component package data, and obtain component metadata;

[0078] S304: Determine the score of each attribute according to the missing value of the field under each attribute in the component metadata;

[0079] S305: Obtain the weight value configured in advance for each attribute, and calculate the comprehensive score of each component by weighted summation;

[0080] S306: receiving input component screening conditions, obtaining all components that meet the component screening conditions, sorting and displaying the components in descending order according to their comprehensive scores; and removing components whose comprehensive scores are less than a preset threshold.

[0081] In the above embodiment, for steps S301 to S303, see Figure 1 The description shown is not repeated here.

[0082] Component recommendation: In the field of software development, it usually refers to an automated service or tool that recommends suitable software components to developers or system integrators based on specific standards or requirements. These recommendations can be based on a variety of factors, such as component performance, compatibility, license requirements, security, maintenance status, or community support. Component recommendation helps simplify the development process, improve development efficiency and software quality. However, the existing component metadata collection solutions cannot be directly connected or integrated with the actual usage scenarios of component recommendations, and cannot recommend and score components through component metadata.

[0083] For steps S304 to S305, this solution calculates the score after setting the weight ratio of multiple metadata (author, code repository information, security vulnerabilities, licenses, etc.) collected by open source components. Specifically, a score is set for each attribute in advance, and the score is adjusted according to the missing field value.

[0084] For example, Name: 10 points, Author: 10 points, License: 15 points, Version: 10 points, Dependencies: 10 points. Only some examples are given here. In fact, there should be multiple, including component package name, component version number, component tag information, component dependency information, component license information, component security vulnerability information, and component source code related information. Set score adjustment rules: 1) If the field value is missing, the score of the attribute is halved. 2) If the field value is an empty string or invalid, it is treated as missing. 3) If the field value is complete and valid, the original score remains unchanged.

[0085] Suppose there is a component metadata including: Name: MyAwesomeLib, Author: XX, License: MIT, Version: 1.0.0, Dependency: Missing. Calculate the score: Name is complete, keep 10 points. Author is complete, keep 10 points. License is complete, keep 15 points. Version is complete, keep 10 points. Dependency is missing, halved to 5 points.

[0086] Assume that the weights of name, author, license, version number, and dependency are configured in advance as 0.1, 0.2, 0.2, 0.2, and 0.3. The final comprehensive score is: Total score = 10*0.1+ 10*0.2 + 15*0.2 + 10*0.2 + 5*0.3 = 9.5 points. In this example, the total score is 11 points. This scoring strategy can help quickly evaluate the metadata quality of components and guide which components to focus on first to improve the integrity and availability of their information.

[0087] For step S306, the user can input component screening conditions as needed, such as one or more of component type, language type, and version number. First, all components that meet the component screening conditions are determined, assuming they are ABCD, and the comprehensive scores of the four are 9, 9.5, 8, and 6 respectively. The components are sorted and displayed in descending order according to their comprehensive scores to obtain BACD. Assuming that the preset threshold is set to 7, D needs to be removed, and only BAC is displayed in the end.

[0088] In the method provided in the above embodiment, ordinary component metadata collection only displays or only supports a single type of component collection, with fewer data indicators. This solution sets multiple collection indicators, including language type, dependency, license integrity, security vulnerabilities, source code related information, etc., so as to screen out more excellent, safe and healthy components based on the missing degree of this information and the score adjustment rules. Compared with the existing technology, it has great advantages and solves the problem that components cannot be recommended through component metadata.

[0089] See also Figure 4, shows a schematic diagram of a collection framework of a specific component metadata according to an embodiment of the present invention, including the following steps:

[0090] 1. A specific acquisition tool is used to obtain component package data from each component package source of the component library or the index file of each component package source;

[0091] 2. Message queue cluster, used to provide data consumption pipeline, receive component package data collected by specific collection tools, and receive component package data obtained by specific consumption tools for consumption;

[0092] 3. Specific consumption tools are responsible for consuming messages from the message queue cluster and calling component label collection service, component dependency collection service, component license collection service, component vulnerability service, and component warehouse service to obtain component label information, component dependency information, component warehouse information, component license information, and component vulnerability information.

[0093] Here, data verification and formatting have been performed during data collection. For data patching and updating services, it is used to determine the vulnerability repair strategy pre-configured for the security vulnerability type according to the security vulnerability type in the security vulnerability information, and use the vulnerability repair strategy to repair the security vulnerability information.

[0094] 4. Component metadata storage service, including distributed database and OSS, stores each component metadata in the distributed database and stores each component package data in the object storage service;

[0095] 5. Component metadata API service, which is used to provide external capabilities such as component query, component addition, deletion, and modification, and component online and offline update through the API interface;

[0096] 6. SBOM document generation service: When generating SBOM documents, if the metadata fields of the collected open source components are missing, the component metadata API service can be queried to complete the component data;

[0097] 7. Component health query service, which is used to calculate the comprehensive score after setting the weight ratio of multiple metadata collected by open source components.

[0098] See also Figure 5 , shows a schematic diagram of main modules of a component metadata collection and processing device 500 provided in an embodiment of the present invention, including:

[0099] The acquisition module 501 is used to determine the package source of each component in the component library, and obtain component package data from each component package source or an index file of each component package source according to a specific acquisition tool;

[0100] The transmission module 502 is used to send each component package data to the message queue cluster through the data consumption pipeline provided by the message queue cluster;

[0101] The consumption module 503 is used to obtain each component package data from the message queue cluster through the data consumption pipeline according to a specific consumption tool, obtain information of specific attributes from each component package data, and obtain component metadata.

[0102] In the implementation device of the present invention, the transmission module 502 is used to: determine the component type according to the component package name in each component package data; according to the correspondence between the preset component type and the topic, send each component package data to the target topic corresponding to the component type in the message queue cluster through the data consumption pipeline provided by the message queue cluster;

[0103] The consumption module 503 is used to obtain each component package data from the target topic through the data consumption pipeline.

[0104] In the implementation device of the present invention, the transmission module 502 is further used to: generate a component package name list based on the component package name, and read the component package data in batches according to the component package names in the component package name list;

[0105] Through the data consumption pipeline provided by the message queue cluster, the read component package data is sent in batches to the topic of the message queue cluster; and

[0106] Update the component package name reading position in the component package name list.

[0107] In the implementation device of the present invention, the specific consumption tool includes one or more of a component tag collection service, a component dependency collection service, a component license collection service, a component vulnerability service, and a component warehouse service;

[0108] The consumption module 503 is used to:

[0109] According to the component label collection service, the language type is determined according to the component package name in each component package data, and each component package data is labeled using a label pre-configured for the language type to obtain component label information;

[0110] Based on the component dependency collection service, the component type is determined according to the component package name, and then the dependency relationship between component packages is determined according to the component package name or component type, and a component dependency tree is constructed according to the dependency relationship to obtain component dependency information;

[0111] Obtain component license information from a component package source of each component package data according to the component license collection service;

[0112] According to the component vulnerability service, a component access link is obtained from the component package data, a page corresponding to the component access link is displayed, security vulnerability information of the component package has been obtained, and security vulnerability information of other component packages that have a dependency relationship with the component package is obtained;

[0113] According to the component repository service, relevant information of the source code of the component package source is obtained from the repository;

[0114] The component metadata is obtained based on one or more of the component package name, component tag information, component dependency information, component license information, component security vulnerability information, and component source code related information.

[0115] In the implementation device of the present invention, the device also includes a storage module, which is used to:

[0116] Store each component metadata in a distributed database and store each component package data in an object storage service.

[0117] In the implementation device of the present invention, the component package data and the component metadata also include a version number, and the device also includes a document module for:

[0118] In the process of generating the software bill of materials document, each field of each component metadata collected is checked for missing field values;

[0119] For target component metadata with missing field values, obtain the target component package name and target version number from the target component metadata;

[0120] Calling the component metadata interface service to determine the component metadata set corresponding to the target component package name from the distributed database;

[0121] Determine the historical version component metadata that is before the target version number in the component metadata collection, obtain the field value corresponding to each field from the historical version component metadata, and supplement the field value to the missing field value position of the software bill of materials document.

[0122] The implementation device of the present invention also includes a component recommendation module, which is used to: determine the score of each attribute based on the missing field value under each attribute in the component metadata; obtain the weight value pre-configured for each attribute, and calculate the comprehensive score of each component using a weighted summation method; receive input component filtering conditions, obtain all components that meet the component filtering conditions, and sort and display them in order from large to small according to the component's comprehensive score; and remove components whose comprehensive scores are less than a preset threshold.

[0123] In addition, the specific implementation content of the device described in the embodiment of the present invention has been described in detail in the method described above, so the repeated content will not be described again here.

[0124] Figure 6 An exemplary system architecture 600 to which embodiments of the present invention may be applied is shown, including terminal devices 601 , 602 , 603 , a network 604 and a server 605 (only an example).

[0125] Terminal devices 601, 602, 603 can be various electronic devices with display screens and supporting web browsing, and various communication client applications are installed. Users can use terminal devices 601, 602, 603 to interact with server 605 through network 604 to receive or send messages, etc.

[0126] The network 604 is used to provide a medium for communication links between the terminal devices 601, 602, 603 and the server 605. The network 604 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0127] The server 605 may be a server that provides various services, such as a backend management server that provides support for shopping websites browsed by users using terminal devices 501, 502, and 503 (for example only). The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - for example only) to the terminal device. It should be noted that the method provided in the embodiment of the present invention is generally executed by the server 605, and accordingly, the device is generally set in the server 605.

[0128] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0129] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 of a terminal device suitable for implementing an embodiment of the present invention. Figure 7 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0130] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0131] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0132] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0133] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0134] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0135] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor. For example, they may be described as follows: a processor includes an acquisition module, a transmission module, and a consumption module. The names of these modules do not, in some cases, constitute limitations on the modules themselves. For example, the consumption module may also be described as a "metadata acquisition module."

[0136] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device executes any of the above component metadata collection and processing methods.

[0137] The computer program product of the present invention comprises a computer program, and when the computer program is executed by a processor, the computer program implements the component metadata collection and processing method in the embodiment of the present invention.

[0138] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for collecting and processing component metadata, characterized in that: include: Determine the package source of each component in the component library, and obtain component package data from each component package source or an index file of each component package source according to a specific acquisition tool; Each component package data is sent to the message queue cluster through the data consumption pipeline provided by the message queue cluster; According to a specific consumption tool, each component package data is obtained from the message queue cluster through the data consumption pipeline, and information of specific attributes is obtained from each component package data to obtain component metadata.

2. The method according to claim 1, characterized in that The data consumption pipeline provided by the message queue cluster sends each component package data to the message queue cluster, including: Determine the component type according to the component package name in each component package data; According to the correspondence between the preset component type and the topic, each component package data is sent to the target topic corresponding to the component type in the message queue cluster through the data consumption pipeline provided by the message queue cluster; The obtaining each component package data from the message queue cluster through the data consumption pipeline includes: Each component package data is obtained from the target topic through the data consumption pipeline.

3. The method according to claim 2, characterized in that The method further comprises: Generate a component package name list based on the component package name, and read component package data in batches according to the component package names in the component package name list; Through the data consumption pipeline provided by the message queue cluster, the read component package data is sent in batches to the topic of the message queue cluster; and The component package name reading position in the component package name list is updated.

4. The method according to claim 1, characterized in that: The specific consumption tool includes one or more of a component tag collection service, a component dependency collection service, a component license collection service, a component vulnerability service, and a component warehouse service; The obtaining of information of specific attributes from each component package data to obtain component metadata includes: According to the component label collection service, the language type is determined according to the component package name in each component package data, and each component package data is labeled using a label pre-configured for the language type to obtain component label information; According to the component dependency collection service, the component type is determined according to the component package name, and then the dependency relationship between component packages is determined according to the component package name or the component type, and a component dependency tree is constructed according to the dependency relationship to obtain component dependency information; Acquiring component license information from a component package source of each component package data according to the component license acquisition service; According to the component vulnerability service, a component access link is obtained from the component package data, a page corresponding to the component access link is displayed, security vulnerability information of the component package is obtained, and security vulnerability information of other component packages that have a dependency relationship with the component package is obtained; According to the component warehouse service, obtaining relevant information of the source code of the component package source from the warehouse; The component metadata is obtained based on one or more of the component package name, component tag information, component dependency information, component license information, component security vulnerability information, and component source code related information.

5. The method according to claim 1 or 4, characterized in that: After obtaining the component metadata, the method further includes: Store each component metadata in a distributed database and store each component package data in an object storage service.

6. The method according to claim 5, characterized in that The component package data and component metadata also include a version number. After obtaining the component metadata, the method further includes: In the process of generating the software bill of materials document, each field of each component metadata collected is checked for missing field values; For target component metadata with missing field values, obtain the target component package name and target version number from the target component metadata; Calling a component metadata interface service to determine a component metadata set corresponding to the target component package name from a distributed database; Determine the historical version component metadata that is before the target version number in the component metadata set, obtain the field value corresponding to each field from the historical version component metadata, and supplement the field value to the missing field value position of the software bill of materials document.

7. The method according to claim 1 or 4, characterized in that: After obtaining the component metadata, the method further includes: Determine the score of each attribute based on the missing field value of each attribute in the component metadata; Obtain the weight value configured in advance for each attribute, and calculate the comprehensive score of each component by weighted summation; Receive input component screening conditions, obtain all components that meet the component screening conditions, sort and display them in descending order according to the comprehensive scores of the components; and remove components whose comprehensive scores are less than a preset threshold.

8. A component metadata collection and processing device, characterized in that: include: A collection module, used to determine the package source of each component in the component library, and obtain component package data from each component package source or an index file of each component package source according to a specific collection tool; The transmission module is used to send each component package data to the message queue cluster through the data consumption pipeline provided by the message queue cluster; The consumption module is used to obtain each component package data from the message queue cluster through the data consumption pipeline according to a specific consumption tool, obtain information of specific attributes from each component package data, and obtain component metadata.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images