Web-based Gitlab topic statistical method, system and equipment and storage medium

Abstract

The invention relates to the technical field of data processing, and particularly provides a web-based Gitlab topic statistical method, system and device and a storage medium, and the method comprises the steps: obtaining login information and a tag ID of a Gitlab application, and writing a mapping formed by the login information and the tag ID into a configuration file; crawling topic data from a web page based on the login information, storing the topic data in a database, and establishing a corresponding relationship between the topic data and the login information; and according to the mapping and the corresponding relationship, establishing an association relationship between the tag ID and the topic data, and generating a topic statistical chart for the tag ID based on the association relationship. According to the invention, full-process automation from data acquisition to analysis and display is realized, and efficient and accurate data support is provided for project management.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and specifically relates to a web-based GitLab topic statistics method, system, device and storage medium. Background Art

[0002] GitLab, a popular code hosting platform, uses its Issues module for project task management and issue tracking during software development. Existing technologies primarily use GitLab's native interface to filter and tally issues, requiring developers to manually switch between tabs to view issue status or rely on limited API calls to retrieve structured data.

[0003] The existing solution has two significant flaws: first, the statistical dimension is single, which can only display basic quantity and status information and lacks the ability to cross-analyze multiple tags; second, dynamic updates are delayed. When the tag system is adjusted or the status of the topic changes, the full query needs to be re-initiated, resulting in poor real-time performance and consumption of system resources. Summary of the Invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a web-based GitLab topic statistics method, system, device and storage medium to solve the above-mentioned technical problems.

[0005] In a first aspect, the present invention provides a web-based GitLab topic statistics method, comprising: Get the login information and tag ID of the GitLab application, and write the mapping between the login information and the tag ID into the configuration file; crawling topic data from a web page based on the login information, saving the topic data to a database, and establishing a corresponding relationship between the topic data and the login information; An association relationship between the tag ID and the topic data is established according to the mapping and the corresponding relationship, and a topic statistical chart is generated for the tag ID based on the association relationship.

[0006] In an optional embodiment, obtaining the login information and tag ID of the GitLab application and writing the mapping consisting of the login information and the tag ID into the configuration file includes: When the GitLab application starts, it calls the GitLab API to obtain the tag ID and user list of the current project, and extracts the login information from the user list.

[0007] In an optional embodiment, the table structure of the database includes: Topic data table, used to store the crawled raw data; Tag table, used to store the mapping relationship between tag ID and readable tag; Login information table, used to store registrant information and login information; The association table is used to store the association between readable tags and topic data.

[0008] In an optional embodiment, crawling topic data from a web page based on the login information, saving the topic data to a database, and establishing a corresponding relationship between the topic data and the login information includes: Retrieve login information from the configuration file; Crawl issue data from the GitLab application's web page based on login information; Saving the crawled topic data to the topic data table, wherein the topic data includes login information; Storing the correspondence between the login information and the topic data table information in the login information table, wherein the topic data table information includes metadata of the table storing the topic data; Establishing a correspondence between the tag ID and the topic data table information according to the mapping in the configuration file and the correspondence between the login information and the topic data table information; Convert the tag ID into a readable tag according to the tag table, and obtain the association relationship between the readable tag and the topic data table information; The association relationship between the readable label and the topic data table information is stored in the association table.

[0009] In an optional embodiment, the method further comprises: Obtain user behavior data; Counting the hot tags with the most queries based on the user behavior data; The topic data associated with the hot tag is retrieved from the database, and the retrieved topic data is stored in the cache.

[0010] In an optional embodiment, establishing an association relationship between the tag ID and the topic data according to the mapping and the corresponding relationship, and generating a topic statistical chart for the tag ID based on the association relationship includes: Get the target tag and determine whether it is a hot tag: If so, generate a topic statistics chart based on the cached corresponding topic data; If not, the corresponding topic data is retrieved from the database to generate a topic statistical chart.

[0011] In an optional embodiment, the corresponding topic data is retrieved from the database to generate a topic statistical chart, including: Query the target topic data table information associated with the target tag from the association table of the database; Read the target topic data from the corresponding target topic data table according to the target topic data table information; Use the front-end drawing tools to generate statistical charts for target topic data.

[0012] In a second aspect, the present invention provides a web-based GitLab topic statistics system, comprising: The basic configuration module is used to obtain the login information and tag ID of the GitLab application and write the mapping between the login information and the tag ID into the configuration file; A data acquisition module, configured to crawl topic data from a web page based on the login information, save the topic data to a database, and establish a corresponding relationship between the topic data and the login information; The topic statistics module is used to establish an association relationship between the tag ID and the topic data according to the mapping and the corresponding relationship, and generate a topic statistics chart for the tag ID based on the association relationship.

[0013] According to a third aspect, a device is provided, comprising: Storage, used to store the web-based GitLab issue statistics program; A processor is used to implement the steps of the web-based Gitlab topic statistics method provided in the first aspect when executing the web-based Gitlab topic statistics program.

[0014] In a fourth aspect, a computer-readable storage medium is provided, on which a web-based Gitlab issue statistics program is stored. When the web-based Gitlab issue statistics program is executed by a processor, the steps of the web-based Gitlab issue statistics method provided in the first aspect are implemented.

[0015] The beneficial effects of the present invention are that the web-based GitLab topic statistics method, system, device, and storage medium provided by the present invention avoid the tedious manual statistics operation by automatically crawling and storing GitLab topic data, significantly improving statistical efficiency; establishing a dynamic association between tag IDs and topic data, ensuring that statistical results are updated in real time as tags change, solving the data lag problem of traditional methods; and automatically generating visual charts based on the association relationship, providing intuitive data display, helping teams quickly grasp project progress. This method realizes full process automation from data collection to analysis and display, providing efficient and accurate data support for project management.

[0016] In addition, the present invention has a reliable design principle, a simple structure and a very broad application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention.

[0019] Figure 2 is another schematic flow chart of a method according to an embodiment of the present invention.

[0020] Figure 3 FIG. 4 is a schematic block diagram of a system according to an embodiment of the present invention.

[0021] Figure 4 A schematic structural diagram of a device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0024] The web-based Gitlab topic statistics method provided by the embodiment of the present invention is executed by a computer device. Accordingly, the web-based Gitlab topic statistics system runs in the computer device.

[0025] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention. Figure 1 The execution entity can be a web-based GitLab issue statistics system. Depending on different needs, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0026] like Figure 1 As shown, the method includes: S1. Obtain the GitLab application login information and tag ID, and write the mapping consisting of the login information and tag ID into the configuration file; S2 crawls topic data from the web page based on the login information, saves the topic data to the database, and establishes a corresponding relationship between the topic data and the login information; S3. Establish an association relationship between the tag ID and the topic data according to the mapping and the corresponding relationship, and generate a topic statistical chart for the tag ID based on the association relationship.

[0027] In an embodiment of the present invention, based on step S1, a possible embodiment will be given below to illustrate its specific implementation scheme in a non-limiting manner.

[0028] When the GitLab application starts, it calls the GitLab API to obtain the tag ID and user list of the current project, and extracts the login information from the user list. Specifically, it includes: Preparation: Read configuration information from environment variables, including the GitLab server address, project ID, and access token; set the HTTP request header to include the authentication token information.

[0029] Get project tags: Construct the API request URL for getting project tags; send an HTTP GET request to the GitLab API; check the response status and throw an exception if the request fails; parse the response JSON data and extract the tag list.

[0030] Get project members: Build the API request URL for getting project members; send an HTTP GET request to the GitLab API; check the response status and throw an exception if the request fails; parse the response JSON data and extract the member list.

[0031] Extract user login name: traverse the member list; extract the username (username field) from each member object; collect all usernames into a list Application startup execution process: When the program starts, perform tag acquisition and member acquisition operations in sequence; print the number of acquired tags and user login names; use the try-except block to capture and handle exceptions that may occur during the API request process.

[0032] In an embodiment of the present invention, based on step S2, a possible embodiment will be given below to illustrate its specific implementation scheme in a non-limiting manner.

[0033] The table structure of the database includes: topic data table, which is used to store the crawled raw data; tag table, which is used to store the mapping relationship between tag ID and readable tag; login information table, which is used to store registrant information and login information; association table, which is used to store the association relationship between readable tags and topic data.

[0034] For example: 1. Issue Data Table (IssueData) Function: Stores the original issue data crawled from GitLab and preserves the original appearance of the data.

[0035] Core fields: issue_id (primary key): unique identifier of GitLab issue; title: issue title; description: issue description; created_at: creation time; updated_at: update time; closed_at: closing time; state: issue status (open / closed); author_id: creator ID (associated with login information table); assignee_ids: assignee ID list (associated with login information table); raw_data: raw JSON data (stores complete API response).

[0036] Index optimization: Create a composite index of created_at and state to speed up queries by time and state; create foreign key indexes for author_id and assignee_ids.

[0037] 2. Tag Mapping Function: Maintain the mapping between GitLab tag IDs and human-readable tag names.

[0038] Core fields: tag_id (primary key): unique identifier of the GitLab tag; tag_name: readable tag name; color: tag color; description: tag description.

[0039] Data source: obtained through the / projects / :id / labels interface of the GitLab API.

[0040] Update mechanism: Full synchronization when the application starts; set up scheduled tasks (such as every morning) for incremental updates.

[0041] 3. Login information table (UserLogin) Function: Stores the mapping between GitLab user ID and login information.

[0042] Core fields: user_id (primary key): unique identifier of GitLab user; username: login username; name: user's full name; email: user's email address; avatar_url: avatar URL; created_at: user creation time.

[0043] Data association: Associate the author_id and assignee_ids in the issue data table through the user_id; obtain it through the / projects / :id / members / all interface of the GitLab API.

[0044] 4. Relation table (IssueTagRelation) Function: Establish a many-to-many association between topics and readable tags.

[0045] Core fields: id (primary key): auto-increment unique identifier; issue_id (foreign key): issue_id of the associated issue data table; tag_name (foreign key): tag_name of the associated tag table; created_at: associated creation time.

[0046] Index optimization: Create a composite unique index for issue_id and tag_name to prevent duplicate associations; create a separate index for tag_name to speed up tag searches.

[0047] Methods for acquiring and storing topic data include: S201. Retrieve login information from the configuration file.

[0048] Configuration file format: Use YAML / JSON format to store sensitive information.

[0049] gitlab: url:https: / / gitlab.com, private_token:<YOUR_PRIVATE_TOKEN> , project_id:123456database: host:localhost, port:5432, user:postgres, password: <password>, dbname:gitlab_issues.

[0050] Security measures: Use environment variables to override sensitive information in configuration files; add configuration files to the version control ignore list (.gitignore). Use encryption tools (such as AnsibleVault) to encrypt configuration files.

[0051] S202. Crawl topic data based on login information.

[0052] Authentication method: Add the PRIVATE-TOKEN field to the HTTP request header; Alternative: OAuth2.0 authorization code mode.

[0053] Paging processing: Use page and per_page parameters to implement paging; Automatically detect the total number of pages and recursively obtain all data.

[0054] Request optimization: Use asynchronous HTTP client (aiohttp / httpx) to send requests in parallel; Set a reasonable request interval (such as 1 second / time) to avoid triggering current limiting; Implement an exponential backoff retry mechanism to handle temporary failures.

[0055] S203. Save the original topic data.

[0056] Data conversion: Map the JSON data returned by the API to database fields; extract nested fields (such as author and assignees) as user IDs.

[0057] Batch insert: accumulate a certain number of records (such as 100) and then insert them in batches; use database transactions to ensure data consistency.

[0058] Conflict handling: Use INSERT...ONCONFLICTUPDATE to handle duplicate data; record data versions and preserve historical changes.

[0059] S204. Store the login information correspondence.

[0060] Deduplication: Check whether the user ID already exists before inserting; if it exists, update it; otherwise, insert it.

[0061] Data association: extract all involved user IDs from the topic data; Batch query login information table and establish mapping relationship; Incremental updates: Record the last synchronization timestamp; Only newly added or modified user information is synchronized.

[0062] S205. Establish a corresponding relationship between the tag ID and the topic.

[0063] Data extraction: Extract the tag ID list from the original topic data; query the corresponding tag name from the tag table.

[0064] Association logic: traverse the tag ID list of each topic; search for readable tag names by tag ID; construct a (topic ID, tag name) tuple.

[0065] S206. Convert the tag ID into a readable tag.

[0066] Cache mechanism: Maintain a mapping cache of tag ID to tag name in memory; set a cache expiration time (such as 30 minutes).

[0067] Batch conversion: Collect all tag IDs to be converted; query the tag table at one time for conversion.

[0068] Exception handling: Log any IDs for which no corresponding tag name is found; by default, use the original tag ID as a fallback.

[0069] S207. Store the association between the readable tag and the topic.

[0070] Batch insert: write all (issue ID, tag name) tuples into the associated table in batches; use the database's multi-value insert syntax to optimize performance.

[0071] Data validation: Verify the validity of the topic ID and tag name; check whether the association relationship already exists.

[0072] Index maintenance: rebuild the index of the associated table after insertion; regularly clean up invalid associations (such as deleted topics).

[0073] In an embodiment of the present invention, based on step S3, a possible embodiment will be given below to illustrate its specific implementation scheme in a non-limiting manner.

[0074] S301. Obtain user behavior data; count the hot tags with the most queries based on the user behavior data; retrieve topic data associated with the hot tags from the database, and store the retrieved topic data in the cache.

[0075] 1. User behavior data collection.

[0076] Front-end tracking: add event monitoring to operations such as tag search and topic browsing; Backend log: records the tag parameters in the API request.

[0077] Data format: json {"user_id":"12345", "action":"tag_search", "tag_name":"bug", "timestamp":"2023-06-15T10:30:00Z"}.

[0078] Storage method: real-time writing to message queues (such as Kafka) and batch importing into analytical databases (such as ClickHouse).

[0079] 2. Calculation of hot tags.

[0080] Statistical period: Real-time statistics: the hot tag list is updated every 5 minutes; Periodic statistics: Statistics of historical hot spots by day / week / month.

[0081] Calculation logic: sql SELECTtag_name,COUNT(*)asquery_countFROMuser_action_logWHEREaction='tag_search' ANDtimestamp>=NOW()-INTERVAL'1DAY'GROUPBYtag_nameORDERBYquery_countDESCLIMIT10.

[0082] Weighting Factor: Recent behavior weighting: Queries in the last hour are weighted three times as much as those from a day ago. Operation type weighting: Advanced search has twice the weight of normal browsing.

[0083] 3. Caching strategy Cache selection: Redis cluster: stores topic data associated with hot tags; Local caching: Use GuavaCache to cache small data sets that are accessed frequently.

[0084] Cache structure: #Redis key design KEY:"hot_tags:{tag_name}:issues" VALUE:[ {"issue_id":123,"title":"System crash issue","state":"open"}, {"issue_id":124,"title":"Performance Optimization Suggestions","state":"closed"}].

[0085] Cache Update: Scheduled task: refresh hotspot tag cache every 15 minutes; Trigger update: When the issue associated with a label changes.

[0086] S302. Obtain a target tag, and determine whether the target tag is a hot tag.

[0087] 1. Label receiving method Front-end delivery: URL parameter: / issuestag=bug; Request body: POST request carries a tag list in JSON format; Format verification: Check if the tag exists in the tag table; Convert tag aliases (e.g. "bugfix" → "bug").

[0088] 2. Hotspot tag determination Data structure: RedisSet: stores the current hotspot tag set; #Redis operation example defis_hot_tag(tag_name): returnredis_client.sismember("hot_tags:set",tag_name).

[0089] Cache penetration processing: When the tag is not in the hotspot set, record the access log; Temporarily cache frequently accessed non-hotspot tags.

[0090] S303. If the target tag is a hot tag, a topic statistics chart is generated based on the cached corresponding topic data; 1. Cache data reading.

[0091] Redis read: defget_hot_issues(tag_name): key=f"hot_tags:{tag_name}:issues" issues=redis_client.get(key), returnjson.loads(issues)ifissueselse[].

[0092] Data verification: Check whether the cached data is expired (TTL>0); Verify data integrity (such as whether required fields are included).

[0093] 2.Generate statistical charts.

[0094] Data preprocessing: Group by time: count the number of topics per week / month; Status classification: Calculate the proportion of different statuses (open / closed).

[0095] Chart Type: Trend chart: shows the change in the number of topics under a tag over time; Pie chart: shows the distribution ratio of topics in different statuses; Histogram: Compare the activity of different tags.

[0096] 3. Front-end rendering.

[0097] Technology selection: Chart.js - a lightweight charting library that supports multiple chart types. ECharts: powerful, supports complex interactions and large amounts of data; Rendering process: / / Chart.js example constctx=document.getElementById('issueChart').getContext('2d');newChart(ctx,{ type:'line', data:{ labels:dates, datasets:[{ label:'OpenIssues', data:openCounts, borderColor:'rgb(75,192,192)', }]}}).

[0098] S304. If the target tag does not belong to a hot tag, the corresponding topic data is retrieved from the database to generate a topic statistical chart.

[0099] (1) Query the target topic data table information that has an association relationship with the target tag from the association table of the database.

[0100] Step-by-step query: Step 1: Query the associated table to obtain the issue ID SELECT issue_id FROM issue_tag_relation WHERE tag_name = 'bug' LIMIT 100; Step 2: Batch query issue details SELECT * FROM issue_data WHERE issue_id IN (123, 124, ...).

[0101] Index optimization: Create a B-Tree index on the tag_name field of the association table; create a primary key index on the issue_id field of the issue table.

[0102] (2) Read the target topic data from the corresponding target topic data table according to the target topic data table information.

[0103] Each query does not exceed 100 records; The front end implements infinite scroll loading; Query current limit: Limit the query frequency of a single tag (e.g. 10 times / minute); Degrade cold label queries.

[0104] (3) Use the front-end drawing tools to generate statistical charts for the target topic data.

[0105] Short-term caching: Query results for non-hot tags are cached for 10 minutes.

[0106] defcache_cold_tag(tag_name,issues): key=f"cold_tags:{tag_name}:issues" redis_client.setex(key,600,json.dumps(issues)).

[0107] LRU elimination strategy: When the cache memory is insufficient, the cold label cache that has not been used for the longest time is eliminated first.

[0108] Please refer to Figure 2 , using a web application to implement this, by grabbing the Issues data from the GitLab webpage, saving it to the database, and then statistically analyzing the data by different dimensions (registrant, label grouping, etc.) and displaying it in the form of charts. The specific description is as follows: 1. Log in using your Gitlab account; 2. After successful login, you can use the browser developer tools to obtain the session ID after login; 3. Configure the mapping relationship between sessionid, labelid and name, the registrant ID to be counted and other related information into the configuration file; 4. Start the web application; 5. Use Python to crawl data, grab Gitlab web page data, and save it in the SQLite database; 6. Query the database data according to different dimensions such as business system functional modules, issue status, and assignees; 7. The front end uses Echarts' pie charts, bar charts, etc. to display the statistical data.

[0109] These include the following technical means: 1. Login authentication and Session management.

[0110] 1. Code logic of the login process.

[0111] Get the CSRF token: Send a stateless GET request to the login page; extract the _gitlab_session value from the response cookie as the initial token.

[0112] Form submission: Construct form data containing username, password and CSRF token; send a POST request to the login interface.

[0113] Session verification: Check whether the response content contains the "dashboard" keyword; if it does, the login is successful, and the complete session ID is extracted and saved.

[0114] 2.Session management logic.

[0115] Encrypted storage: The session ID is encrypted using the AES-256 algorithm; the encryption key comes from the system environment variable.

[0116] Validity verification: Request the user information interface before each crawl; check the response status code and content integrity.

[0117] Automatic refresh: Set a validity period of 2 hours; when half of the validity period has passed, the re-login process will be automatically triggered.

[0118] 2. Configuration file parsing and data mapping.

[0119] 1. Parsing of YAML configuration files.

[0120] Hierarchical parsing: read top-level nodes (gitlab, database, crawler) and parse nested nodes (such as labels and assignees).

[0121] Type conversion: decrypt the string type sessionid into plain text and convert the numeric type configuration item into an integer.

[0122] Validation logic: Checks the existence of required fields, verifies URL format and ID validity.

[0123] 2. Label mapping processing.

[0124] Mapping construction: Read the mapping from tag name to ID from the configuration file and build a bidirectional dictionary for fast query.

[0125] Alias ​​processing: supports defining tag aliases in configuration files and automatically converting them to standard tags during data processing.

[0126] 3. Web data crawling and analysis.

[0127] 1. Crawling process control.

[0128] Concurrency strategy: Use asynchronous coroutines to implement concurrent requests and limit the maximum number of concurrent requests to prevent being banned.

[0129] Exception handling: Automatically retry when network exception occurs (exponential backoff strategy), set request timeout (default 10 seconds).

[0130] Request interval: Randomize the request interval (0.5-2 seconds) and record the request time to prevent excessive frequency.

[0131] 2. Data analysis logic.

[0132] HTML parsing: Use XPath / CSS selectors to locate target elements; process dynamically loaded content (wait for JS rendering to complete).

[0133] Data extraction: extract text content from DOM nodes; extract metadata from attributes (such as date format conversion).

[0134] Data cleaning: remove HTML tags and special characters, and handle null values ​​and abnormal data.

[0135] 4. SQLite database operations.

[0136] 1. Data model mapping.

[0137] Object-relational mapping: Map the crawled dictionary objects to database tables and handle many-to-many relationships (labels, assignees).

[0138] Transaction management: Batch operations use database transactions to ensure data consistency and integrity.

[0139] 2. Data insertion logic.

[0140] Deduplication: Use INSERTORIGNORE to handle unique key conflicts and perform update operations on existing records.

[0141] Batch insert: Perform a batch insert after accumulating 100 records, and use parameterized queries to prevent SQL injection.

[0142] 5. Multi-dimensional statistical query.

[0143] 1. Statistics by business module.

[0144] Query logic: Connect the topic table and tag table through the intermediate table, filter specific business module tags, and count the number of groups by tag.

[0145] Result processing: Convert the query results into a dictionary list and calculate the percentage and ranking information.

[0146] 2. Statistics by topic status.

[0147] Query logic: Directly group the topic table by the status field and count the number of records in each status.

[0148] Data enhancement: Calculate the proportion of each state and generate trend analysis data (changes in the past 7 days).

[0149] 3. Statistics by assignee.

[0150] Query logic: Connect the topic table and the assignee table through an intermediate table, and count the number of assigned topics by assignee.

[0151] Sorting rules: Default sorting is in descending order by quantity, and sorting by name pinyin is supported.

[0152] 6. Front-end data visualization.

[0153] 1. Data interface logic.

[0154] API implementation: Define a RESTful interface to return statistical data, supporting paging, filtering, and sorting parameters.

[0155] Caching strategy: Cache static data for 10 minutes and use weak caching (validation-based) for dynamic data.

[0156] 2. Echarts rendering process.

[0157] Data preparation: Convert the backend JSON data into Echarts format and calculate additional data required for the chart (such as sum and ratio).

[0158] Chart configuration: Set the chart type, title and axis, configure color scheme and animation effects.

[0159] Interaction implementation: Add mouse hover prompt information to realize click-to-drill function.

[0160] 7. System optimization and exception handling.

[0161] 1. Performance optimization logic.

[0162] Data caching: The results are stored in Redis during the first query, and subsequent requests are obtained from the cache first.

[0163] Precomputation: Precomputation tasks are executed every morning and the results are stored in materialized views.

[0164] 2. Exception handling process.

[0165] Crawl exceptions: Automatic retry in case of network timeout (up to 3 times), automatic re-authentication when login fails.

[0166] Database exception: try to reconnect when the connection fails, transaction rollback mechanism.

[0167] Front-end exception: When chart rendering fails, the default chart is displayed and the skeleton screen is used to provide the loading status.

[0168] In some embodiments, the web-based Gitlab topic statistics system may include multiple functional modules composed of computer program segments. The computer program of each program segment in the web-based Gitlab topic statistics system may be stored in the memory of a computer device and executed by at least one processor to perform (see Figure 1 Description) Web-based GitLab issue statistics function.

[0169] In this embodiment, the web-based Gitlab topic statistics system can be divided into multiple functional modules according to the functions it performs, such as Figure 3 As shown. The module referred to in the present invention refers to a series of computer program segments that can be executed by at least one processor and can perform fixed functions, which are stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0170] The basic configuration module is used to obtain the login information and tag ID of the GitLab application and write the mapping between the login information and the tag ID into the configuration file; A data acquisition module, configured to crawl topic data from a web page based on the login information, save the topic data to a database, and establish a corresponding relationship between the topic data and the login information; The topic statistics module is used to establish an association relationship between the tag ID and the topic data according to the mapping and the corresponding relationship, and generate a topic statistics chart for the tag ID based on the association relationship.

[0171] Figure 4 The web-based Gitlab topic statistics method provided for the embodiment of the present application can be applied to a device. Those skilled in the art will understand that the device structure involved in the embodiment of the present invention does not constitute a limitation on the device, and the device may include more or fewer components than shown in the figure, or combine certain components, or arrange different components. In an embodiment of the present invention, the device includes but is not limited to a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present application described and / or required herein.

[0172] The device 400 may include a processor 410, a memory 420, and a communication unit 430. These components communicate via one or more buses. Those skilled in the art will appreciate that the server structure shown in the figure does not limit the present invention. The server structure may be a bus structure or a star structure, and may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0173] Memory 420 can be used to store execution instructions of processor 410. Memory 420 can be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk. When the execution instructions in memory 420 are executed by processor 410, device 400 can perform some or all of the steps in the following method embodiments.

[0174] The processor 410 is the control center of the storage device, which uses various interfaces and lines to connect various parts of the entire electronic device. It runs or executes software programs and / or modules stored in the memory 420, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of multiple packaged ICs with the same or different functions. For example, the processor 410 can only include a central processing unit (CPU). In an embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.

[0175] The communication unit 430 is configured to establish a communication channel so that the storage device can communicate with other devices, receive user data sent by other devices, or send user data to other devices.

[0176] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program that, when executed, may include some or all of the steps of each embodiment provided herein. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0177] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software and a necessary general-purpose hardware platform. Based on this understanding, the technical solutions in the embodiments of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, among other media capable of storing program code, and includes instructions for causing a computer device (which can be a personal computer, a server, or a second device, a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention.

[0178] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment.

[0179] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or modules, and can be electrical, mechanical or other forms.

[0180] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected to achieve the purpose of the present embodiment according to actual needs.

[0181] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0182] Although the present invention has been described in detail with reference to the accompanying drawings and in conjunction with preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, persons of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and such modifications or substitutions shall be within the scope of the present invention. Any changes or substitutions that can be easily conceived by persons skilled in the art within the technical scope disclosed in the present invention shall be within the scope of protection of the present invention.< / password>

Claims

1. A web-based Gitlab topic statistics method, characterized by: include: Get the login information and tag ID of the GitLab application, and write the mapping between the login information and the tag ID into the configuration file; crawling topic data from a web page based on the login information, saving the topic data to a database, and establishing a corresponding relationship between the topic data and the login information; An association relationship between the tag ID and the topic data is established according to the mapping and the corresponding relationship, and a topic statistical chart is generated for the tag ID based on the association relationship.

2. The method according to claim 1, characterized in that Get the GitLab application login information and tag ID, and write the mapping between the login information and tag ID into the configuration file, including: When the GitLab application starts, it calls the GitLab API to obtain the tag ID and user list of the current project, and extracts the login information from the user list.

3. The method according to claim 1, characterized in that The database table structure includes: Topic data table, used to store the crawled raw data; Tag table, used to store the mapping relationship between tag ID and readable tag; Login information table, used to store registrant information and login information; The association table is used to store the association between readable tags and topic data.

4. The method according to claim 3, characterized in that Crawling topic data from a web page based on the login information, saving the topic data to a database, and establishing a corresponding relationship between the topic data and the login information, including: Retrieve login information from the configuration file; Crawl issue data from the GitLab application's web page based on login information; Saving the crawled topic data to the topic data table, wherein the topic data includes login information; Storing the correspondence between the login information and the topic data table information in the login information table, wherein the topic data table information includes metadata of the table storing the topic data; Establishing a correspondence between the tag ID and the topic data table information according to the mapping in the configuration file and the correspondence between the login information and the topic data table information; Convert the tag ID into a readable tag according to the tag table, and obtain the association relationship between the readable tag and the topic data table information; The association relationship between the readable label and the topic data table information is stored in the association table.

5. The method according to claim 1, wherein The method further comprises: Obtain user behavior data; Counting the hot tags with the most queries based on the user behavior data; The topic data associated with the hot tag is retrieved from the database, and the retrieved topic data is stored in the cache.

6. The method according to claim 5, characterized in that Establishing an association relationship between the tag ID and the topic data according to the mapping and the corresponding relationship, and generating a topic statistical chart for the tag ID based on the association relationship, including: Get the target tag and determine whether it is a hot tag: If so, generate a topic statistics chart based on the cached corresponding topic data; If not, the corresponding topic data is retrieved from the database to generate a topic statistical chart.

7. The method according to claim 6, characterized in that Retrieve the corresponding topic data from the database to generate topic statistics charts, including: Query the target topic data table information associated with the target tag from the association table of the database; Read the target topic data from the corresponding target topic data table according to the target topic data table information; Use the front-end drawing tools to generate statistical charts for target topic data.

8. A web-based Gitlab topic statistics system, characterized by: include: The basic configuration module is used to obtain the login information and tag ID of the GitLab application and write the mapping between the login information and the tag ID into the configuration file; A data acquisition module, configured to crawl topic data from a web page based on the login information, save the topic data to a database, and establish a corresponding relationship between the topic data and the login information; The topic statistics module is used to establish an association relationship between the tag ID and the topic data according to the mapping and the corresponding relationship, and generate a topic statistics chart for the tag ID based on the association relationship.

9. A device, characterized in that include: Storage, used to store the web-based GitLab issue statistics program; A processor, configured to implement the steps of the web-based Gitlab topic statistics method according to any one of claims 1 to 7 when executing the web-based Gitlab topic statistics program.

10. A computer-readable storage medium storing a computer program, characterized in that: The readable storage medium stores a web-based Gitlab topic statistics program, which, when executed by a processor, implements the steps of the web-based Gitlab topic statistics method according to any one of claims 1 to 7.

Images