Web-based gitlab issue statistics method, system, device and storage medium

Abstract

The application relates to the technical field of data processing, and specifically provides a web-based Gitlab issue statistical method, system, device and storage medium, which comprises the following steps: obtaining login information and a label ID of a Gitlab application, and writing a mapping composed of the login information and the label ID into a configuration file; crawling issue data from a web page based on the login information, saving the issue data into a database, and establishing a corresponding relationship between the issue data and the login information; establishing an association relationship between the label ID and the issue data according to the mapping and the corresponding relationship, and generating an issue statistical chart for the label ID based on the association relationship. The application realizes full-process automation from data collection to analysis and display, and provides efficient and accurate data support for project management.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically relating to a web-based GitLab topic statistics method, system, device, and storage medium. Background Technology

[0002] In software development, GitLab, as a mainstream code hosting platform, frequently uses its Issues module for project task management and issue tracking. Current technology primarily relies on GitLab's native interface for issue filtering and statistics, requiring developers to manually switch between different tabs to view issue status or depend on limited API calls to obtain structured data.

[0003] The existing solution has two significant drawbacks: First, it has a single statistical dimension, which can only display basic quantity and status information and lacks the ability to perform cross-analysis with multiple labels; second, it is lagging in dynamic updates. When the label system is adjusted or the status of the issue changes, a full query must be re-initiated, resulting in poor real-time performance and consumption of system resources. Summary of the Invention

[0004] In view of the above-mentioned shortcomings of the prior art, the present invention provides a web-based GitLab topic statistics method, system, device and storage medium to solve the above-mentioned technical problems.

[0005] In a first aspect, the present invention provides a web-based GitLab topic statistics method, comprising: Obtain the login information and tag ID of the GitLab application, and write the mapping between the login information and the tag ID into the configuration file; Based on the login information, issue data is crawled from web pages, the issue data is saved to the database, and a correspondence between the issue data and the login information is established. Establish a relationship between tag IDs and issue data based on the mapping and the correspondence, and generate issue statistical charts for tag IDs based on the relationship.

[0006] In one optional implementation, the login information and tag ID of the GitLab application are obtained, and the mapping between the login information and the tag ID is written to a configuration file, including: When the GitLab application starts, it calls the GitLab API to obtain the tag ID and user list of the current project, and extracts the login information from the user list.

[0007] In one optional implementation, the database table structure includes: The topic data table is used to store the raw data crawled. The tag table stores the mapping relationship between tag IDs and readable tags; The login information table is used to store registrant information and login information; The association table is used to store the relationships between readable tags and issue data.

[0008] In an optional implementation, topic data is crawled from a webpage based on the login information, the topic data is saved to a database, and a correspondence between the topic data and the login information is established, including: Retrieve login information from configuration file; Crawl topic data from the GitLab application's web page based on login information; The crawled topic data is saved to the topic data table, and the topic data includes login information; The correspondence between the login information and the topic data table information is stored in the login information table, and the topic data table information includes the metadata of the table storing the topic data; Based on the mapping in the configuration file and the correspondence between the login information and the topic data table information, establish the correspondence between the tag ID and the topic data table information; Based on the label table, the label IDs are converted into readable labels to obtain the association between the readable labels and the issue data table information; The association between readable tags and issue data table information is stored in the association table.

[0009] In an optional implementation, the method further includes: Obtain user behavior data; Based on the user behavior data, the most frequently queried trending tags are statistically analyzed. Retrieve topic data associated with trending tags from the database and store the retrieved topic data in the cache.

[0010] In an optional implementation, an association between tag IDs and issue data is established based on the mapping and the correspondence, and issue statistical charts are generated for tag IDs based on the association, including: Obtain the target tag and determine whether the target tag belongs to the hot tags category: If so, generate topic statistics charts based on the cached topic data; If not, retrieve the corresponding topic data from the database to generate topic statistical charts.

[0011] In one optional implementation, retrieving relevant topic data from a database to generate topic statistical charts includes: Retrieve information from the database's relational tables regarding target topics that are related to the target tags; Read the target issue data from the corresponding target issue data table based on the information in the target issue data table; Use the front-end drawing tools to generate statistical charts for the target topic data.

[0012] Secondly, the present invention provides a web-based GitLab topic statistics system, comprising: The basic configuration module is used to obtain the login information and tag ID of the GitLab application, and write the mapping between the login information and the tag ID into the configuration file; The data acquisition module is used to crawl topic data from web pages based on the login information, save the topic data to the database, and establish a correspondence between the topic data and the login information; The issue statistics module is used to establish the association between tag IDs and issue data according to the mapping and the correspondence, and generate issue statistics charts for tag IDs based on the association.

[0013] Thirdly, a device is provided, comprising: Storage for storing web-based GitLab issue statistics programs; A processor for implementing the steps of the web-based GitLab issue statistics method as provided in the first aspect when executing the web-based GitLab issue statistics program.

[0014] Fourthly, a computer-readable storage medium is provided, on which a web-based GitLab topic statistics program is stored, wherein when the web-based GitLab topic statistics program is executed by a processor, the steps of the web-based GitLab topic statistics method provided in the first aspect are implemented.

[0015] The beneficial effects of this invention are as follows: The web-based GitLab topic statistics method, system, device, and storage medium provided by this invention automatically crawl and store GitLab topic data, avoiding the tedious operation of manual statistics and significantly improving statistical efficiency; it establishes a dynamic association between tag IDs and topic data, ensuring that statistical results are updated in real time as tags change, solving the problem of data lag in traditional methods; and it automatically generates visual charts based on the association, providing intuitive data display and helping teams quickly grasp project progress. This method achieves full-process automation from data collection to analysis and display, providing efficient and accurate data support for project management.

[0016] Furthermore, the design principle of this invention is reliable, the structure is simple, and it has a very wide range of application prospects. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic flowchart of a method according to an embodiment of the present invention.

[0019] Figure 2 This is another illustrative flowchart of a method according to an embodiment of the present invention.

[0020] Figure 3 This is a schematic block diagram of a system according to an embodiment of the present invention.

[0021] Figure 4 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention. Detailed Implementation

[0022] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0024] The web-based GitLab issue statistics method provided in this embodiment of the invention is executed by a computer device, and correspondingly, the web-based GitLab issue statistics system runs on the computer device.

[0025] Figure 1 This is a schematic flowchart illustrating a method according to an embodiment of the present invention. Wherein, Figure 1 The implementing entity can be a web-based GitLab issue statistics system. Depending on different needs, the order of the steps in this flowchart can be changed, and some can be omitted.

[0026] like Figure 1 As shown, the method includes: S1. Obtain the login information and tag ID of the GitLab application, and write the mapping between the login information and the tag ID into the configuration file; S2. Based on the login information, crawl the topic data from the web page, save the topic data to the database, and establish the correspondence between the topic data and the login information; S3. Establish the association between tag ID and issue data according to the mapping and the correspondence, and generate issue statistical charts for tag ID based on the association.

[0027] In one embodiment of the present invention, based on step S1, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0028] When a GitLab application starts, it calls the GitLab API to retrieve the tag IDs and user list of the current project, and then extracts login information from the user list. Specifically, this includes: Preparation: Read configuration information from environment variables, including GitLab server address, project ID, and access token; set HTTP request headers to include authentication token information.

[0029] Get project tags: Construct the API request URL to get project tags; send an HTTP GET request to GitLabAPI; check the response status, and throw an exception if the request fails; parse the response JSON data and extract the tag list.

[0030] Retrieve Project Members: Construct the API request URL to retrieve project members; send an HTTP GET request to GitLabAPI; check the response status, and throw an exception if the request fails; parse the response JSON data and extract the member list.

[0031] Extracting user login names: Iterate through the member list; extract the username (username field) from each member object; collect all usernames into a list. Application startup execution flow: When the program starts, the tag retrieval and member retrieval operations are executed sequentially; the number of tags and user login names retrieved are printed; and a try-except block is used to catch and handle exceptions that may occur during API requests.

[0032] In one embodiment of the present invention, based on step S2, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0033] The database's table structure includes: an issue data table, used to store the raw data crawled; a tag table, used to store the mapping relationship between tag IDs and readable tags; a login information table, used to store registrant information and login information; and an association table, used to store the association relationship between readable tags and issue data.

[0034] For example: 1. Issue Data Table Function: Stores raw topic data crawled from GitLab, preserving the original data appearance.

[0035] Core fields: issue_id (primary key): Unique identifier for a GitLab issue; title: Issue title; description: Issue description; created_at: Creation time; updated_at: Update time; closed_at: Closing time; state: Issue status (open / closed); author_id: Creator ID (related to the login information table); assignee_ids: List of assignee IDs (related to the login information table); raw_data: Raw JSON data (stores the complete API response).

[0036] Index optimization: Create a combined index of created_at and state to speed up queries by time and state; create foreign key indexes for author_id and assignee_ids.

[0037] 2. Tag Mapping Function: Maintains the mapping relationship between GitLab tag IDs and readable tag names.

[0038] Core fields: tag_id (primary key): a unique identifier for a GitLab tag; tag_name: a readable tag name; color: the tag color; description: the tag description.

[0039] Data source: Obtained via the GitLab API's / projects / :id / labels interface.

[0040] Update mechanism: Full synchronization when the application starts; incremental updates via scheduled tasks (e.g., every morning).

[0041] 3. Login Information Table (UserLogin) Function: Stores the mapping between GitLab user IDs and login information.

[0042] Core fields: user_id (primary key): a unique identifier for a GitLab user; username: the login username; name: the user's full name; email: the user's email address; avatar_url: the avatar URL; created_at: the user's creation time.

[0043] Data association: The author_id and assignee_ids in the topic data table are associated with user_id; this is obtained through the GitLab API's / projects / :id / members / all interface.

[0044] 4. Relationship Table (IssueTagRelation) Function: Establishes a many-to-many relationship between topics and readable tags.

[0045] Core fields: id (primary key): auto-incrementing unique identifier; issue_id (foreign key): associated with the issue_id table; tag_name (foreign key): associated with the tag_name table; created_at: associated with the creation time.

[0046] Index optimization: Create a composite unique index for issue_id and tag_name to prevent duplicate associations; create a separate index for tag_name to speed up tag search.

[0047] Methods for acquiring and storing topic data include: S201. Retrieve login information from configuration file.

[0048] Configuration file format: Use YAML / JSON format to store sensitive information.

[0049] gitlab: URL: https: / / gitlab.com private_token:<YOUR_PRIVATE_TOKEN> , project_id:123456database: host:localhost, port:5432 user:postgres, password: <password>, dbname:gitlab_issues.

[0050] Security measures: Use environment variables to override sensitive information in configuration files; add configuration files to a version control ignore list (.gitignore). Encrypt configuration files using encryption tools (such as AnsibleVault).

[0051] S202. Crawl topic data based on login information.

[0052] Authentication method: Add the PRIVATE-TOKEN field to the HTTP request header; Alternative solution: OAuth2.0 authorization code mode.

[0053] Pagination: Implement pagination using the page and per_page parameters; Automatically detect the total number of pages and recursively retrieve all data.

[0054] Request for optimization: Send requests in parallel using an asynchronous HTTP client (aiohttp / httpx); Set a reasonable request interval (e.g., 1 second / time) to avoid triggering rate limiting; Implement an exponential backoff and retry mechanism to handle temporary failures.

[0055] S203. Save the original issue data.

[0056] Data transformation: Map the JSON data returned by the API to database fields; extract nested fields (such as author, assignees) as user IDs.

[0057] Batch insertion: Insert records in batches after accumulating a certain number (e.g., 100); use database transactions to ensure data consistency.

[0058] Conflict handling: Use INSERT...ONCONFLICTUPDATE to handle duplicate data; record data versions and retain historical changes.

[0059] S204. Store the login information correspondence.

[0060] Deduplication: Before inserting, check if the user ID already exists; if it exists, update it; otherwise, insert it.

[0061] Data correlation: Extract all relevant user IDs from the issue data; Batch query the login information table and establish mapping relationships; Incremental update: Record the last synchronization timestamp; Only newly added or modified user information is synchronized.

[0062] S205. Establish the correspondence between tag IDs and topics.

[0063] Data extraction: Extract a list of tag IDs from the original topic data; query the corresponding tag names from the tag table.

[0064] Association logic: Iterate through the list of tag IDs for each topic; find the readable tag name by tag ID; construct a (topic ID, tag name) tuple.

[0065] S206. Convert the tag ID to a readable tag.

[0066] Caching mechanism: Maintain a mapping cache from tag ID to tag name in memory; set a cache expiration time (e.g., 30 minutes).

[0067] Batch conversion: Collect all tag IDs to be converted; query the tag table in one go to perform the conversion.

[0068] Error handling: Logs are logged for IDs where no corresponding tag name is found; the original tag ID is used as a backup by default.

[0069] S207. Store the association between readable tags and topics.

[0070] Batch Insert: Write all (issue ID, tag name) pairs to the associated table in batches; optimize performance using the database's multi-value insertion syntax.

[0071] Data validation: Validate the validity of the issue ID and tag name; check whether the association relationship already exists.

[0072] Index maintenance: Rebuild indexes on related tables after insertion; periodically clean up invalid associations (such as deleted items).

[0073] In one embodiment of the present invention, based on step S3, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0074] S301. Obtain user behavior data; count the most frequently queried hot topics based on the user behavior data; retrieve the topic data associated with the hot topics from the database, and store the retrieved topic data in the cache.

[0075] 1. User behavior data collection.

[0076] Front-end event tracking: Add event listeners to operations such as tag search and topic browsing; Backend logs: Record tag parameters in API requests.

[0077] Data format: json {"user_id":"12345", "action":"tag_search", "tag_name":"bug", "timestamp":"2023-06-15T10:30:00Z"}.

[0078] Storage methods: real-time writing to message queues (such as Kafka), and batch import into analytical databases (such as ClickHouse).

[0079] 2. Hotspot tag calculation.

[0080] Statistical period: Real-time statistics: The list of trending tags is updated every 5 minutes; Periodic statistics: Historical hot topics are statistically analyzed on a daily / weekly / monthly basis.

[0081] Calculation logic: SQL SELECTtag_name,COUNT(*)asquery_countFROMuser_action_logWHEREaction='tag_search' ANDtimestamp>=NOW()-INTERVAL'1DAY'GROUPBYtag_nameORDERBYquery_countDESCLIMIT10.

[0082] Weighting factors: Recent behavior weighting: Queries from the most recent hour have a weight three times that of queries from one day ago; Operation type weighting: Advanced search has twice the weight of normal browsing.

[0083] 3. Caching strategy Cache selection: Redis cluster: Stores topic data associated with hot tags; Local caching: Use GuavaCache to cache frequently accessed small datasets.

[0084] Cache structure: #Redis key design KEY:"hot_tags:{tag_name}:issues" VALUE:[ {"issue_id":123,"title":"System crash issue","state":"open"}, {"issue_id":124,"title":"Performance Optimization Suggestions","state":"closed"}).

[0085] Cache update: Scheduled task: Refresh the hot topic tag cache every 15 minutes; Trigger update: When the issue associated with the tag changes.

[0086] S302. Obtain the target tag and determine whether the target tag belongs to the hot tags.

[0087] 1. Tag Receiving Method Frontend passing: URL parameter: / issuestag=bug; Request body: A POST request carries a list of tags in JSON format; Format validation: Check if the label exists in the label table; Convert tag aliases (e.g., "bugfix" → "bug").

[0088] 2. Hotspot tag identification Data Structures: RedisSet: Stores a collection of currently trending tags; #Example Redis operation defis_hot_tag(tag_name): returnredis_client.sismember("hot_tags:set",tag_name).

[0089] Cache penetration handling: Log access when the tag is not in the hotspot set; Temporarily cache frequently accessed, non-hotspot tags.

[0090] S303. If the target tag is a hot topic tag, then generate a topic statistics chart based on the cached corresponding topic data; 1. Read cached data.

[0091] Redis read: defget_hot_issues(tag_name): key=f"hot_tags:{tag_name}:issues" issues=redis_client.get(key), returnjson.loads(issues)ifissueselse[].

[0092] Data validation: Check if cached data has expired (TTL>0); Verify data integrity (e.g., whether it contains necessary fields).

[0093] 2. Statistical chart generation.

[0094] Data preprocessing: Grouped by time: Statistics on the number of topics per week / month; State classification: Calculate the percentage of different states (open / closed).

[0095] Chart type: Trend chart: Shows the change in the number of topics under a tag over time; Pie chart: Shows the distribution ratio of issues in different states; Bar chart: Compares the activity levels of different tags.

[0096] 3. Front-end rendering.

[0097] Technology selection: Chart.js: A lightweight charting library that supports multiple chart types; ECharts: Powerful features, supports complex interactions and large data volumes; Rendering process: / / Chart.js example constctx=document.getElementById('issueChart').getContext('2d');newChart(ctx,{ type:'line', data:{ labels:dates, datasets:[{ label:'OpenIssues' data:openCounts, borderColor:'rgb(75,192,192)', }]}}).

[0098] S304. If the target tag is not a hot topic tag, retrieve the corresponding topic data from the database to generate a topic statistics chart.

[0099] (1) Query the target topic data table information that is related to the target label from the database association table.

[0100] Step-by-step query: Step 1: Query the related table to get the issue ID: SELECT issue_id FROM issue_tag_relation WHERE tag_name='bug' LIMIT 100; Step 2: Batch query issue details SELECT * FROM issue_data WHERE issue_id IN(123,124,...).

[0101] Index optimization: Create a B-Tree index on the tag_name field of the related table; create a primary key index on the issue_id field of the issue table.

[0102] (2) Read the target issue data from the corresponding target issue data table based on the information in the target issue data table.

[0103] Each query can retrieve no more than 100 records. Implement infinite scrolling loading on the front end; Query rate limiting: Limit the query frequency of a single tag (e.g., 10 times / minute); Degrade cold tag queries.

[0104] (3) Use the front-end drawing tools to generate statistical charts for the target topic data.

[0105] Short-term cache: Cache query results for non-hot tags for 10 minutes.

[0106] defcache_cold_tag(tag_name,issues): key=f"cold_tags:{tag_name}:issues" redis_client.setex(key,600,json.dumps(issues)).

[0107] LRU eviction policy: When cache memory is insufficient, the least used cold tag cache is evicted first.

[0108] Please refer to Figure 2 This is achieved using a web application. Issues data is scraped from a GitLab webpage, saved to a database, and then statistically analyzed according to different dimensions (registrant, label grouping, etc.) and displayed in charts. A detailed description follows: 1. Log in using your GitLab account; 2. After successfully logging in, you can use the browser developer tools to obtain the session ID. 3. Configure the mapping relationship between sessionid, labelid and name, and the relevant information such as the registrant ID that needs to be counted into the configuration file; 4. Start the web application; 5. Use Python to crawl data from GitLab web pages and save it to an SQLite database; 6. Query the database data according to different dimensions such as business system functional modules, topic status, and assignee; 7. The front end uses Echarts' pie charts, bar charts, etc., to display the statistical data.

[0109] This also includes the following technical means: I. Login Authentication and Session Management.

[0110] 1. The code logic for the login process.

[0111] Obtain a CSRF token: Send a stateless GET request to the login page; extract the _gitlab_session value from the response cookie as the initial token.

[0112] Form submission: Construct form data containing username, password, and CSRF token; send a POST request to the login interface.

[0113] Session verification: Check if the response content contains the keyword "dashboard"; if it does, the login is successful, and the complete session ID is extracted and saved.

[0114] 2. Session management logic.

[0115] Encrypted storage: The session ID is encrypted using the AES-256 algorithm; the encryption key comes from system environment variables.

[0116] Validation: Request the user information interface before each crawl; check the response status code and content integrity.

[0117] Automatic refresh: Set a 2-hour validity period; the process of re-login will be automatically triggered when half of the validity period has expired.

[0118] II. Configuration file parsing and data mapping.

[0119] 1. Parsing YAML configuration files.

[0120] Hierarchical parsing: Read the top-level nodes (gitlab, database, crawler) and parse nested nodes (such as labels, assignees).

[0121] Type conversion: Decrypt the string type sessionid into plaintext, and convert the numeric type configuration item into an integer.

[0122] Validation logic: Check if required fields exist, and validate URL format and ID validity.

[0123] 2. Tag mapping processing.

[0124] Mapping construction: Read the mapping from tag names to IDs from the configuration file and build a bidirectional dictionary to enable fast lookup.

[0125] Alias ​​handling: Supports defining tag aliases in the configuration file, which are automatically converted to standard tags during data processing.

[0126] III. Webpage Data Crawling and Parsing.

[0127] 1. Crawling process control.

[0128] Concurrency strategy: Use asynchronous coroutines to implement concurrent requests, and limit the maximum number of concurrent requests to prevent being blocked.

[0129] Error handling: Automatic retry in case of network error (exponential backoff strategy), set request timeout (default 10 seconds).

[0130] Request interval: Randomize the request interval (0.5-2 seconds) and record the request time to prevent excessive frequency.

[0131] 2. Data parsing logic.

[0132] HTML parsing: using XPath / CSS selectors to locate the target element; handling dynamically loaded content (waiting for JS rendering to complete).

[0133] Data extraction: Extracting text content from DOM nodes; extracting metadata from attributes (such as date format conversion).

[0134] Data cleaning: Remove HTML tags and special characters, and handle null values ​​and abnormal data.

[0135] IV. SQLite database operations.

[0136] 1. Data model mapping.

[0137] Object-relational mapping: Maps crawled dictionary objects to database tables, handling many-to-many relationships (tags, assigners).

[0138] Transaction management: Batch operations use database transactions to ensure data consistency and integrity.

[0139] 2. Data insertion logic.

[0140] Deduplication: Use INSERTORIGNORE to handle unique key conflicts and perform update operations on existing records.

[0141] Batch Insert: Perform a batch insert after accumulating 100 records, using parameterized queries to prevent SQL injection.

[0142] V. Multi-dimensional statistical queries.

[0143] 1. Statistics by business module.

[0144] Query logic: Connect the topic table and the tag table through an intermediate table, filter tags for specific business modules, and count the number of items grouped by tag.

[0145] Results processing: Convert the query results into a dictionary list and calculate percentages and ranking information.

[0146] 2. Statistics by issue status.

[0147] Query logic: Directly group the issue table by the status field and count the number of records for each status.

[0148] Data augmentation: Calculate the proportion of each state and generate trend analysis data (changes over the past 7 days).

[0149] 3. Statistics are compiled by the designated person.

[0150] Query logic: Connect the issue table and the assignee table through an intermediate table, and count the number of issues assigned by assignee.

[0151] Sorting rules: By default, sorted in descending order by quantity; sorting by name in pinyin is also supported.

[0152] VI. Front-end data visualization.

[0153] 1. Data interface logic.

[0154] API Implementation: Define a RESTful interface to return statistical data, supporting pagination, filtering, and sorting parameters.

[0155] Caching strategy: Cache static data for 10 minutes, and use weak caching for dynamic data (after verification).

[0156] 2. Echarts rendering process.

[0157] Data preparation: Convert the backend JSON data to Echarts format and calculate the additional data required for the chart (such as sums and proportions).

[0158] Chart configuration: Set the chart type, title, and axes, and configure the color scheme and animation effects.

[0159] Interactive implementation: Add mouse hover tooltip to enable drill-down functionality.

[0160] VII. System Optimization and Anomaly Handling.

[0161] 1. Performance optimization logic.

[0162] Data caching: The results of the first query are stored in Redis, and subsequent requests will retrieve them from the cache first.

[0163] Pre-calculation: A pre-calculation task is executed every day at midnight, and the results are stored in the materialized view.

[0164] 2. Exception handling process.

[0165] Crawling error: Automatic retry due to network timeout (maximum 3 times); automatic re-authentication when login fails.

[0166] Database exception: When the connection fails, an attempt is made to reconnect, and a transaction rollback mechanism is used.

[0167] Front-end error: When the chart fails to render, a default chart is displayed, and a skeleton screen is used to provide the loading status.

[0168] In some embodiments, the web-based GitLab issue statistics system may include multiple functional modules composed of computer program segments. The computer programs for each segment in the web-based GitLab issue statistics system may be stored in the memory of a computer device and executed by at least one processor to perform (see details). Figure 1 (Description) Functionality for web-based GitLab topic statistics.

[0169] In this embodiment, the web-based GitLab topic statistics system can be divided into multiple functional modules based on its functions, such as... Figure 3 As shown. The module referred to in this invention is a series of computer program segments that can be executed by at least one processor and perform a fixed function, and is stored in memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0170] The basic configuration module is used to obtain the login information and tag ID of the GitLab application, and write the mapping between the login information and the tag ID into the configuration file; The data acquisition module is used to crawl topic data from web pages based on the login information, save the topic data to the database, and establish a correspondence between the topic data and the login information; The issue statistics module is used to establish the association between tag IDs and issue data according to the mapping and the correspondence, and generate issue statistics charts for tag IDs based on the association.

[0171] Figure 4 The web-based GitLab topic statistics method provided in the embodiments of this application can be applied to devices. Those skilled in the art will understand that the device structures involved in the embodiments of this invention do not constitute a limitation on the device. A device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. In the embodiments of this invention, the device includes, but is not limited to, laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0172] The device 400 may include a processor 410, a memory 420, and a communication unit 430. These components communicate via one or more buses. Those skilled in the art will understand that the server structure shown in the figure does not constitute a limitation of the present invention. It may be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0173] The memory 420 can be used to store execution instructions of the processor 410. The memory 420 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. When the execution instructions in the memory 420 are executed by the processor 410, the device 400 is able to perform some or all of the steps in the above method embodiments.

[0174] The processor 410 serves as the control center of the storage device, connecting various parts of the electronic device via various interfaces and lines. It executes software programs and / or modules stored in the memory 420, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 410 may consist only of a central processing unit (CPU). In this embodiment of the invention, the CPU may have a single processing core or include multiple processing cores.

[0175] The communication unit 430 is used to establish a communication channel, enabling the storage device to communicate with other devices. It can receive user data sent by other devices or send user data to other devices.

[0176] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, which, when executed, may include some or all of the steps provided in the embodiments of the present invention. The storage medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0177] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or any other medium capable of storing program code. It includes several instructions to cause a computer device (which may be a personal computer, a server, or a second device, network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0178] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0179] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or modules may be electrical, mechanical, or other forms.

[0180] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0181] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0182] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the present invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should also be covered within the protection scope of the present invention.< / password>

Claims

1. A web-based GitLab topic statistics method, characterized in that, include: Obtain the login information and tag ID of the GitLab application, and write the mapping between the login information and the tag ID into the configuration file; Based on the login information, issue data is crawled from web pages, the issue data is saved to the database, and a correspondence between the issue data and the login information is established. Establish an association between tag IDs and issue data based on the mapping and the correspondence, and generate issue statistical charts for tag IDs based on the association. The database table structure includes: The topic data table is used to store the raw data crawled. The tag table stores the mapping relationship between tag IDs and readable tags; The login information table is used to store registrant information and login information; The association table is used to store the relationships between readable tags and issue data; Based on the login information, topic data is crawled from web pages, the topic data is saved to a database, and a correspondence between the topic data and the login information is established, including: Retrieve login information from configuration file; Crawl topic data from the GitLab application's web page based on login information; The crawled topic data is saved to the topic data table, and the topic data includes login information; The correspondence between the login information and the topic data table information is stored in the login information table, and the topic data table information includes the metadata of the table storing the topic data; Based on the mapping in the configuration file and the correspondence between the login information and the topic data table information, establish the correspondence between the tag ID and the topic data table information; Based on the label table, the label IDs are converted into readable labels to obtain the association between the readable labels and the issue data table information; Store the association relationship between readable tags and issue data table information in the association table; The method further includes: Obtain user behavior data; Based on the user behavior data, the most frequently queried trending tags are statistically analyzed. Retrieve topic data associated with trending tags from the database and store the retrieved topic data in the cache; Establish an association between tag IDs and issue data based on the mapping and the correspondence, and generate issue statistical charts for tag IDs based on the association, including: Obtain the target tag and determine whether the target tag belongs to the hot tags category: If so, generate topic statistics charts based on the cached topic data; If not, retrieve the corresponding topic data from the database to generate topic statistical charts.

2. The method according to claim 1, characterized in that, Retrieve the login information and tag ID of the GitLab application, and write the mapping between the login information and the tag ID into a configuration file, including: When the GitLab application starts, it calls the GitLab API to obtain the tag ID and user list of the current project, and extracts the login information from the user list.

3. The method according to claim 1, characterized in that, Retrieve relevant topic data from the database to generate topic statistical charts, including: Retrieve information from the database's relational tables regarding target topics that are related to the target tags; Read the target issue data from the corresponding target issue data table based on the information in the target issue data table; Use the front-end drawing tools to generate statistical charts for the target topic data.

4. A web-based GitLab topic statistics system, characterized in that, include: The basic configuration module is used to obtain the login information and tag ID of the GitLab application, and write the mapping between the login information and the tag ID into the configuration file; The data acquisition module is used to crawl topic data from web pages based on the login information, save the topic data to the database, and establish a correspondence between the topic data and the login information; The issue statistics module is used to establish an association between tag IDs and issue data based on the mapping and the correspondence, and to generate issue statistics charts for tag IDs based on the association. The database table structure includes: The topic data table is used to store the raw data crawled. The tag table stores the mapping relationship between tag IDs and readable tags; The login information table is used to store registrant information and login information; The association table is used to store the relationships between readable tags and issue data; Based on the login information, topic data is crawled from web pages, the topic data is saved to a database, and a correspondence between the topic data and the login information is established, including: Retrieve login information from configuration file; Crawl topic data from the GitLab application's web page based on login information; The crawled topic data is saved to the topic data table, and the topic data includes login information; The correspondence between the login information and the topic data table information is stored in the login information table, and the topic data table information includes the metadata of the table storing the topic data; Based on the mapping in the configuration file and the correspondence between the login information and the topic data table information, establish the correspondence between the tag ID and the topic data table information; Based on the label table, the label IDs are converted into readable labels to obtain the association between the readable labels and the issue data table information; Store the association relationship between readable tags and issue data table information in the association table; The system is also configured to perform the following steps: Obtain user behavior data; Based on the user behavior data, the most frequently queried trending tags are statistically analyzed. Retrieve topic data associated with trending tags from the database and store the retrieved topic data in the cache; Establish an association between tag IDs and issue data based on the mapping and the correspondence, and generate issue statistical charts for tag IDs based on the association, including: Obtain the target tag and determine whether the target tag belongs to the hot tags category: If so, generate topic statistics charts based on the cached topic data; If not, retrieve the corresponding topic data from the database to generate topic statistical charts.

5. A device, characterized in that, include: Storage for storing web-based GitLab issue statistics programs; A processor, configured to implement the steps of the web-based GitLab issue statistics method as described in any one of claims 1-3 when executing the web-based GitLab issue statistics program.

6. A computer-readable storage medium storing a computer program, characterized in that, The readable storage medium stores a web-based GitLab topic statistics program, which, when executed by a processor, implements the steps of the web-based GitLab topic statistics method as described in any one of claims 1-3.

Images