A code data asset management system

Abstract

The present invention provides a code data asset management system, including a code data application layer, a tool platform service layer and a data storage layer. The code data application layer is used to provide code data application services to users; the tool platform service layer is used to collect code data and process it, including: a collection module for collecting multi-source code data from multiple public network platforms in real time, a storage module for storing the collected code data in real time in a preset distributed database or object storage service platform, and a processing module for processing the collected code data. The processing module includes a data deduplication submodule, a license detection submodule, an identity information processing submodule, a data filtering submodule, a field deduplication submodule, a code extraction submodule, a code identification submodule and a data improvement submodule; the data storage layer is used to store the code data processed by the tool platform service layer. The system provided by the present invention can improve the user's code usage efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of Internet technology, and in particular relates to a code data asset management system. Background Art

[0002] In recent years, with the development of large-scale AI models, hardware computing power, and computing frameworks, the computing speed of computers has been able to catch up with the pace of data production. In the service chain based on intelligent computing applications, data processing is the most time-consuming link, often accounting for 70% or more of the entire application service process. After the rise of large language models, the use of large language models to directly understand the unstructured data generated by humans can save time for structuring unstructured data and use computing power to replace some manual work. However, in the practice of training large language models, it is found that low-quality training corpus cannot build an accurate general model. Therefore, an efficient data management system is of great significance.

[0003] In the field of data assets, code data has been ignored because it cannot directly generate economic value. However, code is the product of human advanced thinking. It represents modern intelligent science and computer culture and can be used to build portals, blogs, short video apps, game applications, and intelligent application services. However, the current problem is that code data is very large and redundant, and ordinary data management models are not suitable for code data.

[0004] At present, huggingface, a well-known AI open source community in the United States, hosts the collected code data (The Stack dataset) and specially recruits computer programming practitioners to process these code data. A large part of the work is done manually. In addition, the Github community can host a large number of data warehouses, but it only hosts the data, not manages it. Whether it is the big data bureau or data technology companies in China, there are no products that can realize the collection and processing of code data. Therefore, there is an urgent need to build a system that can realize the collection, effective processing and application of code data. Summary of the invention

[0005] The embodiment of the present invention provides a code data asset management system to solve the problem of the current lack of a system capable of collecting, effectively processing and applying code data.

[0006] In order to solve the above technical problems, the embodiments of the present invention disclose the following technical solutions:

[0007] One aspect of the present invention provides a code data asset management system, the system comprising a code data application layer, a tool platform service layer and a data storage layer, wherein:

[0008] The code data application layer is configured to provide code data application services to users;

[0009] The tool platform service layer is configured to collect code data and process it, including a collection module, a storage module and a processing module, wherein:

[0010] The acquisition module is configured to acquire multiple source code data from multiple public network platforms in real time;

[0011] The storage module is configured to store the collected code data in a preset distributed database or object storage service platform in real time;

[0012] The processing module is configured to process the collected code data, and includes a data deduplication submodule, a license detection submodule, an identity information processing submodule, a data filtering submodule, a field deduplication submodule, a code extraction submodule, a code identification submodule, and a data improvement submodule;

[0013] The data storage layer is configured to store code data processed by the tool platform service layer.

[0014] Optionally, the acquisition module includes a web page submodule, a social media platform submodule, a public database submodule, and a forum and discussion area submodule;

[0015] The data collected by the collection module includes text format data, non-text format data and structured data, among which text format data includes text containing code content, web page code, programming language code and Markdown markup language code; non-text format data includes pictures, video data and audio data containing code content; structured data includes CSV or Excel data containing code content, as well as JSON data.

[0016] Optionally, the data deduplication submodule includes a hash signature calculation unit, a grouping unit and a duplicate item selection unit, wherein:

[0017] A hash signature calculation unit, configured to calculate a MinHashes hash signature for each code data respectively;

[0018] A grouping unit is configured to group all code data according to the local sensitive hash LSH and the MinHashes hash signature of each code data, wherein the code data with a similarity exceeding a preset threshold is assigned to a group;

[0019] The duplicate item selection unit is configured to retain a code data for each group in the following manner:

[0020] Get the importance value of each code data in the same group,

[0021] Determine whether there is a code data with a greater importance value than other code data,

[0022] If it exists, retain the code data;

[0023] If it does not exist, a code data is randomly selected to be retained.

[0024] Optionally, the license detection submodule includes a regular expression unit, a ScanCode unit, a license propagation unit and a license transmission unit, wherein:

[0025] A regular expression unit is configured to use a regular expression to determine, for each code data, whether the code data has a license attribute;

[0026] A ScanCode unit is configured to detect the code data without license attributes using the ScanCode Toolkit open source toolkit, and again determine whether the code data has license attributes;

[0027] A license propagation unit, configured to share the license information contained in the code data with the license attribute with the code data having the same basic path as the code data and not having the license attribute;

[0028] The license classification unit is configured to classify the license of each code data, wherein the license categories include: no license, loose license and strict license.

[0029] Optionally, the identity information processing submodule includes a StarEncoder model unit, a PII model unit and a model training unit, wherein:

[0030] The StarEncoder model unit is configured to build an encoder-only StarEncoder encoder model;

[0031] The PII model unit is configured to add a preceding layer as a token classification header on top of the StarEncoder encoder model, wherein the token classification header includes six target categories, namely, name, email, key, password, IP address, and username; the PII model unit is further configured to adjust the architecture of the StarEncoder encoder model to obtain a PII personal identity information detection model;

[0032] The model training unit is configured to obtain code data with name, email, key, password, IP address and username tags as training data, train the PII personal identity information detection model, and use the trained PII personal identity information detection model to perform identity information detection on all code data.

[0033] Optionally, the data filtering submodule includes a first filtering unit, a second filtering unit and a Jupyter conversion unit, wherein:

[0034] The first filtering unit is configured to, when the code data is obtained from the email data, use a regular expression to delete the text automatically generated when the email replies to the question; and is also configured to, when the code data is obtained from the comment data, delete the question in the short message with less than a preset number of characters, and at the same time, truncate the comment text according to a preset number of lines, and only retain the last preset number of lines in the truncated text; and is also configured to delete the comment text replied by the machine; and is also configured to determine whether there is a question text containing more than 10 events, and if so, delete the question text;

[0035] The second filtering unit is configured to remove code files containing more than 100 characters; is also configured to remove questions or comments with empty submission topics; is also configured to remove questions or comments containing a stop word list; is also configured to remove questions or comments with a space-delimited character ratio of more than 20; is also configured to subsample data in JSON, YAML, XML, and HTML formats with a probability of 50%;

[0036] The Jupyter conversion unit is configured to convert Jupyter notebooks to code format and export Jupyter notebooks to text, code, and output links.

[0037] Optionally, the field deduplication submodule includes a json field processing unit, a rawhtml field removal unit and a duplicate field processing unit, wherein:

[0038] The JSON field processing unit is configured to flatten the JSON fields in each code data to the same level;

[0039] A rawhtml field removal unit is configured to remove the rawhtml field in each code data;

[0040] The repeated field processing unit is configured to use a matching algorithm to identify whether there is one or more groups of repeated fields in each code data, and if so, retain one of each group of repeated fields and delete the remaining repeated fields.

[0041] Optionally, the code extraction submodule includes an information acquisition unit, a target code fragment extraction unit, a storage unit and a processing unit, wherein:

[0042] The information acquisition unit is configured to acquire a source code library path or a preset file provided by a user, and a code snippet extraction rule;

[0043] A target code snippet extraction unit is configured to extract the target code snippet from a source code library path or a preset file based on a regular expression matching method and according to the extraction rule;

[0044] A storage unit configured to store the target code segment as independent code data;

[0045] The processing unit is configured to perform testing or integration processing on the target code fragment.

[0046] Optionally, the code recognition submodule includes a programming language recognition unit, a classification unit and a re-extraction unit, wherein:

[0047] A programming language recognition unit is configured to input the target code snippet into a preset language recognition model or a preset language recognition tool for recognition, and obtain the programming language to which the target code snippet belongs;

[0048] a classification unit configured to classify the target code snippet according to the programming language to which the target code snippet belongs;

[0049] The re-extraction unit is configured to determine whether a re-extraction instruction sent by a user is received, and if so, perform a second extraction on the target code segment and classify the second-extracted code segment.

[0050] Optionally, the data improvement submodule includes a preprocessing unit, a matching unit, an enhancement unit and a cleaning unit, wherein:

[0051] A preprocessing unit, configured to preprocess all question-answer data, wherein the preprocessing at least includes word segmentation, removal of stop words, and removal of punctuation marks;

[0052] A matching unit, configured to use a preset matching model to vectorize the questions and answers in each set of question-answering data, and calculate the similarity between the questions and the answers;

[0053] an enhancement unit, configured to perform enhancement processing on the question and answer data when the similarity is lower than a first preset threshold, wherein the enhancement processing at least includes generating multiple samples according to the question and the answer, and performing synonym replacement on the text in the question and the answer;

[0054] The cleaning unit is configured to clean the questions and answers whose similarity is lower than a second preset threshold.

[0055] The code data asset management system disclosed in the embodiment of the present invention at least includes a code data application layer, a tool platform service layer and a data storage layer, wherein the code data application layer is used to provide code data application services to users; the tool platform service layer is used to collect code data and process it, including a collection module, a storage module and a processing module, wherein the collection module is used to collect multiple source code data from multiple public network platforms in real time; the storage module is used to store the collected code data in real time in a preset distributed database or object storage service platform; the processing module is used to process the collected code data, including a data deduplication submodule, a license detection submodule, an identity information processing submodule, a data filtering submodule, a field deduplication submodule, a code extraction submodule, a code identification submodule and a data improvement submodule; the data storage layer is used to store the code data processed by the tool platform service layer. The system provided by the embodiment of the present invention can realize the collection, effective processing and application of code data, and help and improve the efficiency of user code use.

[0056] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the disclosure, nor is it intended to limit the scope of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present disclosure.

[0058] Figure 1 A schematic diagram of the structure of a code data asset management system provided by an embodiment of the present invention is shown;

[0059] Figure 2 A schematic diagram of the structure of a code data application layer provided by an embodiment of the present invention is shown;

[0060] Figure 3 A schematic diagram of the structure of a tool platform service layer provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0061] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0062] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0063] Figure 1 A schematic diagram of a code data asset management system provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the system includes a code data application layer 1, a tool platform service layer 2 and a data storage layer 3.

[0064] The following is a detailed description of each layer:

[0065] 1. Code Data Application Layer

[0066] The code data application layer 1 is configured to provide code data services to users. In one embodiment disclosed in the present invention, Figure 2 As shown, the code data application layer 1 includes a code generation module 11, a code completion module 12, an intelligent programming IDE module 13, an intelligent operation and maintenance module 14, a code checking module 15, a unit testing module 16 and an automatic function calling module 17.

[0067] The main function of the code generation module 11 is to automatically generate code, reducing the time and effort of developers to write repetitive code. This module automatically generates code according to given specifications based on predefined templates or using machine learning algorithms. For example, when developing a Web application, it can automatically generate access codes for the back-end database or HTML elements for the front-end.

[0068] The code completion module 12 can help developers write code faster and improve coding efficiency through smart prompts and automatic completion of code snippets. It can use contextual information such as variable types, existing code libraries and language specifications to predict and provide possible code completion options.

[0069] The intelligent programming IDE module 13 can provide a feature-rich environment that supports code writing, debugging and management, including syntax highlighting, code completion, code navigation, intelligent refactoring tools, version control integration and other functions, making development more efficient and orderly.

[0070] The intelligent operation and maintenance module 14 can use automation and intelligent technology to monitor and manage the performance and health of software applications and systems, including functions such as fault detection, automatic recovery, resource optimization and security monitoring, to help reduce the burden of manual operation and maintenance work.

[0071] The code checking module 15 is used to automatically analyze the code quality, detect potential programming errors, vulnerabilities, code style issues, etc., and can provide instant feedback during the development process to help developers correct problems and improve the stability and maintainability of the code.

[0072] The unit testing module 16 can automatically test the smallest testable part of the software (usually a function or method) and is integrated into the continuous integration / continuous deployment (CI / CD) process to ensure that code changes do not break existing functions.

[0073] The automatic function calling module 17 can automatically call predefined functions or methods in appropriate contexts to improve development efficiency and code reusability. For example, in object-oriented programming, the corresponding method may be automatically called according to the state of the object to process a request or update the interface.

[0074] 2. Tool Platform Service Layer

[0075] The tool platform service layer 2 is configured to collect and process code data, including three parts of the data processing process: the source of data, processing tools, and data format. First, use crawler tools to obtain data. The data can come from HTML content, PDF documents, and some JavaScript-generated content in web pages, as well as from social media platforms, public databases, forums, and discussion areas. Then, after obtaining the web page content, use web page parsing tools and format conversion tools, in conjunction with the corresponding data cleaning and conversion processes, to convert the data into the desired format, such as text format (text, HTML, programming language code, Markdown format, etc.), non-text format (pictures, video and audio files, etc.), and structured data (tables, JSON files, etc.).

[0076] like Figure 3 As shown, the tool platform service layer 2 includes a collection module 21, a storage module 22 and a processing module 23.

[0077] The acquisition module 21 is configured to collect multi-source code data from multiple public network platforms in real time. In the embodiment disclosed in the present invention, the acquisition module 21 includes a Web page acquisition submodule, a social media platform submodule, a public database submodule, and a forum and discussion area submodule, wherein the Web page acquisition submodule is used to collect HTML content, PDF documents, and JavaScript generated content, etc.; the social media platform submodule is used to collect multi-source data related to code in the social media platform; the public database submodule is used to collect data related to code in the public database; the forum and discussion area submodule is used to collect discussion topic content and question and answer data related to code in the forum and discussion area, etc.

[0078] The data collected by the collection module 21 includes text format data, non-text format data and structured data, wherein the text format data includes text containing code content, web page code, programming language code and Markdown markup language code; the non-text format data includes pictures, video data and audio data containing code content; the structured data includes CSV or Excel data containing code content, as well as JSON data.

[0079] The storage module 22 is configured to store the collected code data in real time in a preset distributed database or object storage service platform.

[0080] The processing module 23 is configured to process the collected code data, including a data deduplication submodule 231, a license detection submodule 232, an identity information processing submodule 233, a data filtering submodule 234, a field deduplication submodule 235, a code extraction submodule 236, a code recognition submodule 237 and a data improvement submodule 238.

[0081] The following is a detailed description of each submodule:

[0082] (1) Data deduplication submodule

[0083] Data deduplication can help reduce redundant information in a data set, thereby saving storage space and improving data processing efficiency. By removing duplicate data records, the data set can be made more compact and efficient. Duplicate data records may lead to a decline in data quality, such as introducing bias during statistical analysis and machine learning model training. The deduplicated data set is more streamlined and data processing is faster, which helps speed up the data processing process and improve user work efficiency.

[0084] The tools or models used in the data deduplication submodule 231 are:

[0085] 1. MinHash+LSH: MinHash+LSH inherits the settings of CodeParrot and is composed of MinHash and locality sensitive hashing (LSH) based on datasketch9, with a tuple (non-alphanumeric tag) and a similarity threshold of 0.85 Jaccard.

[0086] 2. PaLM Coder: is a string metric used to measure the difference between two sequences.

[0087] 3. CodeGen: used to calculate the file checksum sha-256, which can achieve deduplication of code data files.

[0088] The data deduplication submodule 231 first calculates the MinHashes of all code data, and then uses local sensitive hashing (LSH) to group the data according to the MinHash fingerprint of the code data. In the LSH stage, "similar" data files are assigned to the same storage bucket, identifying them as duplicates. Only one data file is selected from each duplicate group to achieve data file deduplication. In addition to the above method, in order to retain the repository context, files from repositories with higher star and fork counts or from the latest submission date can also be prioritized as a tiebreaker. In a specific embodiment disclosed in the present invention, 5-grams and Jaccard similarity of 0.7 can be used.

[0089] Star and fork are indicators of repository popularity on GitHub, an Internet-based code hosting platform and development collaboration tool that provides developers with a range of features that enable them to work together, share code, perform version control, and manage projects. Stars express the user's love and attention to the repository, while forks indicate that other users have created their own branches based on the repository. Therefore, repositories with higher star and fork counts tend to be popular, high-quality projects that may have more user contributions, richer features, and better documentation.

[0090] The latest commit data means the data updated in the most recent commit. In software development, the latest commit usually means updates and improvements to the code or documentation, which may include bug fixes, additions of new features, etc. Therefore, considering the files updated in the most recent commit can ensure that you focus on the latest information and the latest code status, which is very important for tracking the activity and development trends of the repository.

[0091] The data deduplication submodule 231 includes a hash signature calculation unit, a grouping unit, and a duplicate item selection unit, wherein:

[0092] The hash signature calculation unit is configured to calculate the MinHashes hash signature of each code data respectively.

[0093] The grouping unit is configured to group all code data according to the local sensitive hash LSH and the MinHashes hash signature of each code data, wherein code data with a similarity exceeding a preset threshold (ie, very similar code data) is assigned to a group.

[0094] The duplicate item selection unit is configured to retain a code data for each group in the following manner:

[0095] Get the importance value of each code data in the same group, for example, the Star and fork of the code data.

[0096] Determine whether there is a code data with a greater importance value than other code data,

[0097] If present, retain the code data;

[0098] If it does not exist, a code data is randomly selected to be retained.

[0099] (2) License Detection Submodule

[0100] All repository-level license information matching the dataset name is extracted from the code repository. When the repository-level license is not available, the ScanCode Toolkit open source toolkit is used to detect the file-level license.

[0101] The license detection submodule 232 includes a regular expression unit, a ScanCode unit, a license propagation unit and a license transmission unit, wherein:

[0102] The regular expression unit is configured to use a regular expression to determine, for each code data, whether the code data has a license attribute. A regular expression is a tool used to describe a string pattern, and can be used to match, search, and replace a specific pattern or format in a text.

[0103] The ScanCode unit is configured to use the ScanCode Toolkit open source toolkit to detect the code data that does not have the license attribute, and again determine whether the code data has the license attribute.

[0104] The license propagation unit is configured to share the license information contained in the code data with the license attribute with other code data without the license attribute that has the same basic path as the code data.

[0105] The license classification unit is configured to classify the license of each code data, wherein the license categories include: no license, loose license and strict license.

[0106] (3) Identity information processing submodule

[0107] The main reason for identity information processing is to protect personal privacy and data security. By reducing the personal identity information in the redistributed training data, the risk of personal privacy leakage can be effectively reduced, and potential legal liability and loss of trust can be avoided. In addition, compliance requirements are also an important consideration. Many regulations and standards (such as GDPR, HIPAA, etc.) require the protection of users' personal identity information when processing data. Therefore, identity information processing is a key measure in the data processing process, which helps to ensure the compliance and security of data processing.

[0108] The identity information processing submodule 233 includes a StarEncoder model unit, a PII model unit and a model training unit, wherein:

[0109] The StarEncoder model unit is configured to build an encoder-only StarEncoder encoder model.

[0110] The PII model unit is configured to add a preceding layer on top of the StarEncoder encoder model as a token classification header, and the token classification header includes 6 target categories: name, email, key, password, IP address and username; the PII model unit is also configured to adjust the architecture of the StarEncoder encoder model to obtain a PII personal identity information detection model.

[0111] The model training unit is configured to obtain code data with names, email addresses, keys, passwords, IP addresses and user names as training data, train the PII personal identity information detection model, and use the trained PII personal identity information detection model to perform identity information detection on all code data.

[0112] (4) Data filtering submodule

[0113] The data filtering submodule 234 is used to filter and clean up a large amount of code and related data to improve data quality and usability. Through a series of processing steps and filters, the module can effectively identify and handle problems in the data, such as robot replies, short messages, empty submission subjects, etc., thereby purifying the data set and providing a high-quality data foundation for subsequent analysis and processing.

[0114] The disclosed embodiment of the present invention adopts the bigcode-preprocessing preprocessing tool to complete large-scale code preprocessing, and provides the function of processing and cleaning large code bases.

[0115] The data filtering submodule 234 includes a first filtering unit, a second filtering unit and a Jupyter conversion unit, wherein:

[0116] The first filtering unit is configured to use a regular expression to delete the text automatically generated when replying to a question in an email when the code data is obtained from email data; it is also configured to delete the question in the short message with less than a preset number of characters when the code data is obtained from comment data, and at the same time, truncate the comment text according to a preset number of lines and only retain the last preset number of lines in the truncated text; it is also configured to delete the comment text of the machine reply; it is also configured to determine whether there is a question text containing more than 10 events, and if so, delete the question text.

[0117] The first filtering unit can filter Github issues. GitHub Issue is a feature on the GitHub platform used to track tasks, defects, suggestions or other requirements in a project. It is a tool for collaboration and discussion that enables project teams to easily communicate with users, record problems and track the progress of solutions. In GitHub, each project can create one or more Issues to record problems and tasks in the project. Each Issue contains some key information, such as title, description, label, assigned personnel, milestones, etc., to better organize and manage work.

[0118] First (first), regular expressions were used to remove text that was automatically generated when users responded to questions via email. Secondly (second), questions with short messages (less than 200 characters) were removed, and long comments in the middle were truncated to a maximum of 100 lines, while retaining the last 20 lines. Reduced the volume by 18%. Thirdly (third), comments from robots were excluded, and robot keywords were searched in the usernames of comment authors, eliminating 17% of the total events and causing 14.7% of the questions to be cleared. Then (fourth), the number of users participating in the conversation was used as a quality indicator. The standard is to contain conversations with two or more users. However, if a single user wrote a question that contained more than ten events, these issues were excluded because these issues were often of poor quality or originated from neglected robots. By implementing these filters, another 14% of the questions were eliminated. Finally, non-English questions were filtered out using the fasttext library.

[0119] The second filtering unit is configured to remove code files containing more than 100 characters; is also configured to remove questions or comments with empty submission topics; is also configured to remove questions or comments containing a stop word list; is also configured to remove questions or comments with more than 20 characters separated by spaces; and is also configured to subsample data in JSON, YAML, XML, and HTML formats with a probability of 50%.

[0120] The second filter unit filters Git commits, which refer to the submission records in the Git version control system. Each change to the code base can be recorded and saved by committing (commit) to form a submission record. Each submission record contains a description of a set of changes and the corresponding author, timestamp and other information.

[0121] First, remove code files containing or greater than 100k characters. Second, remove commits with empty subject. Third, remove commits containing or equal to the stopword list. Then, remove commits with a space-separated word character ratio of more than 20. Finally, subsample data formats (JSON, YAML, XML, HTML) with a probability of 50%.

[0122] The Jupyter conversion unit is configured to convert Jupyter notebooks to code format and export Jupyter notebooks to text, code, and output links.

[0123] Jupyter is the name of an open source project that provides an interactive computing environment that supports multiple programming languages, including Python, R, Julia, etc. The core component of Jupyter is Jupyter Notebook, a web-based application that allows users to create and share documents containing code, text, images, formulas, etc. These documents are called Notebooks. The code in Jupyter Notebook can be executed interactively, and the output results can be displayed in real time in the Notebook, making it a powerful tool for exploring, visualizing, and sharing data.

[0124] Jupyter Notebook is one of the core components of the Jupyter project. It is a web-based interactive computing environment for creating and sharing documents containing code, text, images, formulas, etc., which are called notebooks.

[0125] Jupyter notebooks are presented in the form of web pages, where users can write and execute code and view code execution results, charts, text output, etc. in real time. This interactive approach allows users to complete data exploration, analysis, visualization, and document writing in the same environment, making it ideal for scenarios such as data science, machine learning, teaching, and report writing.

[0126] Jupyter notebooks support multiple programming languages, the most commonly used of which is Python, but also supports other languages ​​such as R, Julia, Scala, etc. Users can write and execute code in different languages ​​by adding code blocks in the notebook and selecting the corresponding programming language kernel.

[0127] In addition to code blocks, Jupyter notebooks also support text blocks in Markdown format. Users can insert elements such as titles, paragraphs, lists, links, pictures, etc. into documents, as well as insert mathematical formulas using LaTeX syntax.

[0128] Convert Jupyter notebooks to code formats, such as scripts. Usually, Jupyter notebooks contain a lot of text, code, and output results, and converting them to the same format as pure code can remove the text and output and keep only the code part.

[0129] Convert Jupyter to formatted text, code, and output chains. Typically, Jupyter notebooks are presented in a mixed format, including code blocks, Markdown text blocks, and code output. Converting it to a formatted form will organize and lay out these elements according to specific typesetting rules so that they are easier to read and understand. This conversion can be achieved by exporting the notebook to other formats (such as HTML, PDF).

[0130] (5) Field deduplication submodule

[0131] The field deduplication submodule 235 is used to delete duplicate fields and deduplicate fields to obtain a more lightweight data set.

[0132] Although there is a deduplication method in the above embodiment, the deduplication process is operated between different data, and does not take into account the situation where there are repeated fields in a JSON data. Therefore, the embodiment of the present invention uses a preset opencsg algorithm to delete the repeated fields and slim down the data.

[0133] The field deduplication submodule 235 includes a json field processing unit, a rawhtml field removal unit and a duplicate field processing unit, wherein:

[0134] The JSON field processing unit is configured to flatten the JSON fields in each code data to the same level.

[0135] The purpose of this step is to unify the different fields or elements in the JSON data to the same level, no matter how complex their original structure is, to simplify the data structure and make subsequent processing more convenient.

[0136] In JSON data, different objects may have different nesting levels and structures. Some fields may be directly located at the top level, while some fields may be nested inside other objects. In the disclosed embodiments of the present invention, it is desired to analyze, query or process the entire data set without paying attention to the specific structure. Therefore, the JSON fields are flattened to the same level.

[0137] The rawhtml field removal unit is configured to remove the rawhtml field from each code data. Since some data is web crawled data, different field structures are nested, and the rawhtml field removal unit can process the unprocessed fields more finely.

[0138] The repeated field processing unit is configured to use a matching algorithm to identify whether there is one or more groups of repeated fields in each code data, and if so, retain one of each group of repeated fields and delete the remaining repeated fields.

[0139] (6) Code extraction submodule

[0140] The code extraction submodule 236 allows users to extract specific functions or module codes from the source code library or related files for reuse, testing or maintenance, which can help developers effectively manage and organize the code library and improve the maintainability and reusability of the code.

[0141] The embodiment of the present invention uses regular matching to search and identify text fragments that meet specific rules. In the code extraction submodule 236, the regular matching function is mainly used to filter out code fragments that meet the conditions from the source code according to the extraction rules provided by the user.

[0142] The embodiment of the present invention also uses Snippet to extract code. Snippet is a code extraction tool from the github open source community.

[0143] The code extraction submodule 236 includes an information acquisition unit, a target code segment extraction unit, a storage unit and a processing unit, wherein:

[0144] The information acquisition unit is configured to acquire a source code library path or a preset file provided by a user, as well as a code snippet extraction rule.

[0145] The target code snippet extraction unit is configured to extract the target code snippet from the source code library path or the preset file based on the regular expression matching method and according to the extraction rule.

[0146] Extracted code snippets, contain code snippets extracted from the source code repository. The extracted code file contains the extracted code snippets, which can be a single file or multiple files. The specific format depends on the extracted code snippets and the user's needs.

[0147] The storage unit is configured to store the target code segment as independent code data or code module.

[0148] The processing unit is configured to perform testing or integration processing on the target code fragment.

[0149] (7) Code recognition submodule

[0150] The code identification submodule 237 is designed to help users quickly and accurately identify and understand the programming language type of source code. Through this module, users can easily determine the programming language of a given code snippet or file, thereby more effectively managing, organizing and processing code resources. This module provides a variety of tools and models to ensure recognition coverage of various programming languages, and combines advanced algorithms and technologies to provide highly reliable recognition results.

[0151] In the disclosed embodiment of the present invention, the code identification submodule 237 uses the following tools to implement module functions:

[0152] 1. github-linguist / linguist: Determine the programming language based on the file suffix.

[0153] 2. guesslang-go: Written in golang language, an improvement based on guesslang.

[0154] 3. guesslang: Written in Python, supports 54 languages.

[0155] 4. CodeBERTa-language-id: CodeBERTa-language-id is fine-tuned on CodeBERTa-small-v1. CodeBERTa is a RoBERTa-like model trained on GitHub's CodeSearchNet dataset, but only supports 6 programming languages.

[0156] 5. Find the required Java code snippets in a large number of Java source files, and use CodeBERT with abstract trees to automatically classify the code snippets extracted from Code4ML; use multinomial naive Bayes (MNB) classifier; DeepSCC: fine-tuned RoBERTa model to classify the programming language type of the source code (19 types); EL-codebert: add Bi-LSTM layer; Codeprompt: based on EL-codebert, remove the Bi-LSTM layer and add Prompt Wrapper.

[0157] The code recognition submodule 237 includes a programming language recognition unit, a classification unit and a re-extraction unit, wherein:

[0158] The programming language recognition unit is configured to input the target code segment into a preset language recognition model or a preset language recognition tool for recognition, and obtain the programming language to which the target code segment belongs.

[0159] The classification unit is configured to classify the target code snippet according to the programming language to which the target code snippet belongs.

[0160] The re-extraction unit is configured to determine whether a re-extraction instruction sent by a user is received, and if so, perform a second extraction on the target code segment and classify the second-extracted code segment.

[0161] (8) Data completion submodule

[0162] The data improvement submodule 238 is used to solve the mismatch or inconsistency problem in the code data, especially the mismatch between the question and the answer in the question and answer data. Through this module, the data can be further processed and cleaned to ensure the matching and consistency between the question and the answer.

[0163] In the embodiment disclosed in the present invention, the Google word2vec tool is used to convert text into vector representation and capture the semantic relationship between words.

[0164] The data improvement submodule 238 includes a pre-processing unit, a matching unit, an enhancement unit and a cleaning unit, wherein:

[0165] The preprocessing unit is configured to preprocess all question-answer data, and the preprocessing at least includes word segmentation, removal of stop words, and removal of punctuation marks.

[0166] The matching unit is configured to use a preset matching model to vectorize the questions and answers in each set of question-answering data and calculate the similarity between the questions and the answers.

[0167] The enhancement unit is configured to perform enhancement processing on the question and answer data when the similarity is lower than a first preset threshold value, and the enhancement processing at least includes generating multiple samples based on the questions and answers, and performing synonym replacement on the texts in the questions and answers.

[0168] The cleaning unit is configured to clean the questions and answers whose similarity is lower than a second preset threshold.

[0169] In the embodiments disclosed in the present invention, the processing tools used include crawler tools, web page tag parsing tools, data cleaning and conversion tools, and format conversion tools, among which the crawler tools include Scrapy, Selenium, and BeautifulSoup; the web page tag parsing tools include Xpath, Lxml, and Jsoup; the data cleaning and conversion tools include OpenRefine and Talend; the format conversion tools include: Pandoc, html2text, html2md, and PDFMiner / PyPDF2.

[0170] 3. Data Storage Layer

[0171] The data storage layer 3 is configured to store the code data processed by the tool platform service layer 2. In the embodiment disclosed in the present invention, the distributed databases supported by the data storage layer 3 include Hive, HBase, Apache Hadoop HDFS, object storage Amazon S3, Microsoft Azure Data Lake, Google Cloud Storage, etc.

[0172] 4. Infrastructure Application Layer

[0173] In the embodiments disclosed in the present invention, Figure 1 As shown, the code data asset management system also includes an infrastructure application layer 4, which mainly includes a computing unit and a storage unit. The computing unit performs calculations and logical processing operations, wherein the CPU is suitable for task-oriented sequential processing, and the GPU has a more efficient parallel processing capability and is widely used in the fields of image processing and deep learning. The storage unit includes SSD (solid state drive) and NAS (network attached storage) for efficient storage and reading.

[0174] 5. Operating System Layer

[0175] In the embodiments disclosed in the present invention, Figure 1 As shown, the code data asset management system also includes an operating system layer 5, which is adapted to various popular operating systems and a variety of terminal users, such as Windows systems, macOS systems, Linux systems, Unix systems, Android systems, iOS systems, HarmonyOS systems, etc.

[0176] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A code data asset management system, characterized in that: The system includes a code data application layer, a tool platform service layer and a data storage layer, wherein: The code data application layer is configured to provide code data application services to users; The tool platform service layer is configured to collect code data and process it, including a collection module, a storage module and a processing module, wherein: The acquisition module is configured to acquire multiple source code data from multiple public network platforms in real time; The storage module is configured to store the collected code data in a preset distributed database or object storage service platform in real time; The processing module is configured to process the collected code data, and includes a data deduplication submodule, a license detection submodule, an identity information processing submodule, a data filtering submodule, a field deduplication submodule, a code extraction submodule, a code identification submodule, and a data improvement submodule; The license detection submodule includes a regular expression unit, a ScanCode unit, a license propagation unit and a license transmission unit, wherein: A regular expression unit is configured to use a regular expression to determine, for each code data, whether the code data has a license attribute; A ScanCode unit is configured to detect the code data without license attributes using the ScanCode Toolkit open source toolkit, and again determine whether the code data has license attributes; A license propagation unit, configured to share the license information contained in the code data with the license attribute with the code data having the same basic path as the code data and not having the license attribute; A license classification unit is configured to classify the license of each code data, wherein the license categories include: no license, loose license, and strict license; The data filtering submodule includes a first filtering unit, a second filtering unit and a Jupyter conversion unit, wherein: The first filtering unit is configured to, when the code data is obtained from the email data, use a regular expression to delete the text automatically generated when the email replies to the question; and is also configured to, when the code data is obtained from the comment data, delete the question in the short message with less than a preset number of characters, and at the same time, truncate the comment text according to a preset number of lines, and only retain the last preset number of lines in the truncated text; and is also configured to delete the comment text replied by the machine; and is also configured to determine whether there is a question text containing more than 10 events, and if so, delete the question text; The second filtering unit is configured to remove code files containing more than 100 characters; is also configured to remove questions or comments with empty submission topics; is also configured to remove questions or comments containing a stop word list; is also configured to remove questions or comments with a space-delimited character ratio of more than 20; is also configured to subsample data in JSON, YAML, XML, and HTML formats with a probability of 50%; A Jupyter conversion unit configured to convert Jupyter notebooks to code format and export Jupyter notebooks to text, code, and output chains; The data storage layer is configured to store code data processed by the tool platform service layer.

2. The system according to claim 1, characterized in that The acquisition module includes a web page submodule, a social media platform submodule, a public database submodule, and a forum and discussion area submodule; The data collected by the collection module includes text format data, non-text format data and structured data, among which text format data includes text containing code content, web page code, programming language code and Markdown markup language code; non-text format data includes pictures, video data and audio data containing code content; structured data includes CSV or Excel data containing code content, as well as JSON data.

3. The system according to claim 1, characterized in that The data deduplication submodule includes a hash signature calculation unit, a grouping unit and a duplicate item selection unit, wherein: A hash signature calculation unit, configured to calculate a MinHashes hash signature for each code data respectively; A grouping unit is configured to group all code data according to the local sensitive hash LSH and the MinHashes hash signature of each code data, wherein the code data with a similarity exceeding a preset threshold is assigned to a group; The duplicate item selection unit is configured to retain a code data for each group in the following manner: Get the importance value of each code data in the same group, Determine whether there is a code data with a greater importance value than other code data, If it exists, retain the code data; If it does not exist, a code data is randomly selected to be retained.

4. The system according to claim 1, characterized in that The identity information processing submodule includes a StarEncoder model unit, a PII model unit and a model training unit, wherein: The StarEncoder model unit is configured to build an encoder-only StarEncoder encoder model; The PII model unit is configured to add a preceding layer as a token classification header on top of the StarEncoder encoder model, wherein the token classification header includes six target categories, namely, name, email, key, password, IP address, and username; the PII model unit is further configured to adjust the architecture of the StarEncoder encoder model to obtain a PII personal identity information detection model; The model training unit is configured to obtain code data with name, email, key, password, IP address and username tags as training data, train the PII personal identity information detection model, and use the trained PII personal identity information detection model to perform identity information detection on all code data.

5. The system according to claim 1, characterized in that The field deduplication submodule includes a json field processing unit, a rawhtml field removal unit and a duplicate field processing unit, wherein: The JSON field processing unit is configured to flatten the JSON fields in each code data to the same level; A rawhtml field removal unit is configured to remove the rawhtml field in each code data; The repeated field processing unit is configured to use a matching algorithm to identify whether there is one or more groups of repeated fields in each code data, and if so, retain one of each group of repeated fields and delete the remaining repeated fields.

6. The system according to claim 1, characterized in that The code extraction submodule includes an information acquisition unit, a target code segment extraction unit, a storage unit and a processing unit, wherein: The information acquisition unit is configured to acquire a source code library path or a preset file provided by a user, and a code snippet extraction rule; A target code snippet extraction unit is configured to extract the target code snippet from a source code library path or a preset file based on a regular expression matching method and according to the extraction rule; A storage unit configured to store the target code segment as independent code data; The processing unit is configured to perform testing or integration processing on the target code fragment.

7. The system according to claim 6, characterized in that The code recognition submodule includes a programming language recognition unit, a classification unit and a re-extraction unit, wherein: A programming language recognition unit is configured to input the target code snippet into a preset language recognition model or a preset language recognition tool for recognition, and obtain the programming language to which the target code snippet belongs; a classification unit configured to classify the target code snippet according to the programming language to which the target code snippet belongs; The re-extraction unit is configured to determine whether a re-extraction instruction sent by a user is received, and if so, perform a second extraction on the target code segment and classify the second-extracted code segment.

8. The system according to claim 1, characterized in that The data improvement submodule includes a preprocessing unit, a matching unit, an enhancement unit and a cleaning unit, wherein: A preprocessing unit, configured to preprocess all question-answer data, wherein the preprocessing at least includes word segmentation, removal of stop words, and removal of punctuation marks; A matching unit, configured to use a preset matching model to vectorize the questions and answers in each set of question-answering data, and calculate the similarity between the questions and the answers; an enhancement unit, configured to perform enhancement processing on the question and answer data when the similarity is lower than a first preset threshold, wherein the enhancement processing at least includes generating multiple samples according to the question and the answer, and performing synonym replacement on the text in the question and the answer; The cleaning unit is configured to clean the questions and answers whose similarity is lower than a second preset threshold.

Images