A method and device for quickly constructing a code search index
By registering Code Search sharding in the code search tool, optimizing index construction queues and merging tasks, the problem that code search tools in the existing technology are difficult to provide timely and accurate search services in large-scale code bases, and efficient and flexible code retrieval is achieved.
Patent Information
- Application Number
- CN202411632531.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-15
- Publication Date
- 2025-03-11
- Estimated Expiration
- 2044-11-15
AI Technical Summary
When faced with large-scale code bases, existing code search tools are difficult to provide timely and accurate retrieval services, especially in multi-user and high concurrency scenarios, the retrieval experience may be greatly reduced.
By receiving the index task request sent by the search service, the Code Search shard that has not been registered is registered as a valid search shard, and the mold is taken based on the warehouse ID and the number of valid search shards, the association relationship between the warehouse and the shard is recorded, and the index construction queue is obtained. Then, the index build queue order is updated according to the preset priority, and the index build tasks of the same repository are merged according to the preset task merging strategy to obtain the optimized index build queue. The search service builds the queue according to the preset periodic timed polling optimized indexes, and obtains the index construction task, and builds the code index based on the index construction task.
Provide timely and accurate search services, greatly improving the efficiency and flexibility of code retrieval, and maintaining stable and efficient performance when facing large-scale code bases.
Smart Images
Figure CN119166741B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of software development technology, and in particular to a method and device for quickly constructing a code search index. Background Art
[0002] With the rapid development of software development and the increasing size of code bases, developers have an increasing demand for efficient, accurate and real-time code search. However, most code search tools on the market often face problems such as delayed index updates and insufficient scalability when dealing with these needs. Especially when facing large-scale code bases, it is difficult to provide timely and accurate retrieval services.
[0003] Taking GitHub, the world's largest code hosting platform, as an example, after users submit their code to the platform, they usually have to wait for dozens of minutes or even longer to retrieve the newly submitted code in the platform's search function. This delay is difficult to meet the needs of developers in development scenarios with high timeliness requirements. For example, when developers need to quickly find and debug the newly submitted code changes, they cannot retrieve the latest content immediately, which will significantly affect development efficiency.
[0004] This is because platforms such as GitHub usually use batch index updates to process changes in the code base in batches at regular intervals, rather than creating update tasks in real time to synchronize index updates after the repository is updated. Although this approach has certain advantages in resource utilization, it is insufficient in terms of timeliness and flexibility, especially in multi-user, high-concurrency scenarios, the search experience may be greatly reduced. Summary of the invention
[0005] To this end, the present application provides a method and device for quickly constructing a code search index to solve the problem that the code search method existing in the prior art is difficult to provide timely and accurate retrieval services.
[0006] In order to achieve the above objectives, this application provides the following technical solutions:
[0007] In a first aspect, a method for quickly constructing a code search index is provided, the method being applied to an index queue service, comprising:
[0008] Step 1: Receive the indexing task request sent by the search service;
[0009] Step 2: Register the unregistered Code Search shard as a valid search shard according to the index task request;
[0010] Step 3: Perform a modulus operation based on the warehouse ID and the number of valid search shards, and record the association between the warehouse and the shards to obtain an index building queue;
[0011] Step 4: Update the order of the index building queue according to the preset priority, and merge the index building tasks of the same warehouse according to the preset task merging strategy to obtain an optimized index building queue; the search service regularly polls the optimized index building queue according to the preset period, obtains the index building tasks, and builds the code index according to the index building tasks.
[0012] Preferably, in step 1, the index task request includes: the search service address, shard name and disk information.
[0013] Preferably, in step 4, when the order of the index building queue is updated according to the preset priority, specifically: warehouses with large priority weights are placed at the front of the index building queue, and warehouses with consistent weights are sorted according to code update time.
[0014] Preferably, in step 4, when merging the index building tasks of the same warehouse according to the pre-set task merging strategy, specifically: if there are multiple code update tasks, they are merged into one code update task; if there is a new warehouse task and it contains multiple code update tasks, they are merged into one code library new creation task; if there are multiple code update tasks and it contains a warehouse archiving or deletion task, they are merged into one warehouse deletion task.
[0015] Preferably, in step 4, the search service supports concurrent index construction of a single shard and parallel index construction of multiple shards.
[0016] Preferably, in step 4, the search service builds the code index according to the index building task, specifically including: index initialization, new repository, incremental update and index deletion.
[0017] Preferably, a complete index file is constructed during the index initialization and the creation of a new repository, and is stored in the disk of the search service.
[0018] Preferably, the incremental update specifically includes: obtaining a repository that needs to be updated, incrementally pulling the latest version of the repository to a disk for code search, and updating an index file based on the incremental content.
[0019] Preferably, if the search service fails to build the code index according to the index building task, it will first pull the incremental updated code and then execute the index building steps; if it fails again, it will try to delete the cached code directory and re-clone the code of this warehouse in full to ensure successful code download. After the code is pulled successfully, the index building task will be triggered again. If the index incremental update task fails, it will try to delete the index file and re-build the index in full to ensure successful index task construction.
[0020] In a second aspect, a device for quickly constructing a code search index includes:
[0021] An index task request receiving module, used to receive an index task request sent by a search service;
[0022] A registration module, used to register the unregistered Code Search shard as a valid search shard according to the index task request;
[0023] An initialization module is used to perform a modulus operation based on the warehouse ID and the number of valid search shards, and record the association between the warehouse and the shards to obtain an index building queue;
[0024] The index building queue optimization module is used to update the order of the index building queue according to a preset priority, and merge the index building tasks of the same warehouse according to a preset task merging strategy to obtain an optimized index building queue; the search service regularly polls the optimized index building queue according to a preset period, obtains the index building tasks, and builds the code index according to the index building tasks.
[0025] Compared with the prior art, this application has at least the following beneficial effects:
[0026] The present application provides a method and device for quickly building a code search index by receiving an index task request sent by a search service; registering unregistered Code Search shards as valid search shards according to the index task request; taking a modulus based on the warehouse ID and the number of valid search shards, and recording the association between the warehouse and the shards to obtain an index building queue; updating the order of the index building queue according to a preset priority, and merging the index building tasks of the same warehouse according to a preset task merging strategy to obtain an optimized index building queue; the search service periodically polls the optimized index building queue according to a preset period, and obtains the index building task, and builds a code index according to the index building task. The method and device for quickly building a code search index provided by the present application can provide timely and accurate retrieval services, greatly improving the efficiency and flexibility of code retrieval. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] In order to more intuitively illustrate the prior art and the present application, exemplary drawings are given below. It should be understood that the specific shapes and structures shown in the drawings should not generally be regarded as limiting conditions for implementing the present application; for example, those skilled in the art are capable of easily making conventional adjustments or further optimizations to the addition / reduction / attribution division, specific shapes, positional relationships, connection methods, dimensional ratios, etc. of certain units (components) based on the technical concepts and exemplary drawings disclosed in the present application.
[0028] Figure 1A flowchart of a method for quickly constructing a code search index provided in Example 1 of the present application;
[0029] Figure 2 A schematic diagram of the structure of a method for quickly constructing a code search index provided in Example 1 of the present application;
[0030] Figure 3 A schematic diagram of a retry structure after a code index fails to be constructed for the search service provided in Example 1 of the present application. DETAILED DESCRIPTION
[0031] The present application is further described below in detail through specific embodiments in conjunction with the accompanying drawings.
[0032] In the description of this application: unless otherwise specified, "plurality" means two or more. The terms "first", "second", "third", etc. in this application are intended to distinguish the objects referred to, and do not have special meanings in terms of technical connotations (for example, they should not be understood as emphasizing the importance or order, etc.). Expressions such as "including", "comprising", "having", etc. also mean "not limited to" (certain units, components, materials, steps, etc.).
[0033] The terms such as "upper", "lower", "left", "right", "middle", etc. cited in this application are usually used to indicate the general relative position relationship for the purpose of intuitive understanding by referring to the drawings, and are not absolute limitations on the position relationship in the actual product.
[0034] The purpose of this application is to provide a method for quickly building a code search index, so that after the user submits the code to the code hosting platform, the user can quickly search for the newly submitted code content on the platform, thereby improving the timeliness of the user's code search.
[0035] Embodiment 1
[0036] See also Figure 1 and Figure 2 This embodiment provides a method for quickly building a code search index, which is applied to an index queue service and includes:
[0037] S1: Receive the indexing task request sent by the search service;
[0038] Specifically, after the Code Search service (a service used to search and index code repositories) is deployed and started, the search service will initiate a scan to the index queue with its own address (address), shard name (shard_name), and disk information (disk_usage), and send an index task request to the index queue service.
[0039] S2: Register the unregistered Code Search shard as a valid search shard according to the index task request;
[0040] Specifically, after receiving the index task request, if the index queue service finds that the Code Search shard has not been registered, it will automatically register it according to the search service's own address, shard name and disk information, and mark it as a valid search shard. At the same time, Code Search supports rapid horizontal expansion. In the later stage, if the number of warehouses increases to a certain level, more warehouse search requests can be supported by expanding shards.
[0041] S3: Modulo the warehouse ID and the number of valid search shards, and record the association between the warehouse and the shards to obtain the index building queue;
[0042] After the service shards are registered, the index queue service provides an interface for initializing the existing code indexing task, which is responsible for allocating warehouses to each Code Search service and building indexes. Specifically, the index queue service will determine which shard the warehouse index is initialized in for the first time by taking the warehouse ID and the number of Code Search shards (valid search shards). After initialization, the index queue will record the relationship between the warehouse and the shard to ensure that the subsequent index update tasks of the warehouse can be correctly routed to the existing shards.
[0043] The core of index initialization is to generate a batch of index tasks to be built. The Code Search service will scan the tasks in the index building queue according to the set scanning frequency, and parse the acquired tasks by type. It currently supports monitoring events including code push, branch change, and repository change. When the code library triggers the corresponding event, a task will be inserted into the index queue for Code Search to consume.
[0044] S4: Update the order of the index building queue according to a preset priority, and merge the index building tasks of the same warehouse according to a preset task merging strategy to obtain an optimized index building queue; the search service periodically polls the optimized index building queue according to a preset period, obtains the index building tasks, and builds a code index according to the index building tasks.
[0045] Specifically, after the index task to be built is generated, the index queue service will first perform optimization processing according to the preset priority and task merging strategy.
[0046] When optimizing based on priority, warehouses with higher priority weights are placed at the front of the queue, and warehouses with consistent weights are sorted by code update time, ensuring that the code that users focus on can be indexed first.
[0047] When optimizing according to the preset task merging strategy, the update tasks of the same repository will be merged according to the events. If there are multiple code update tasks, they will eventually be merged into one code update task. If there is a task to create a new repository and it contains multiple code update tasks, they will eventually be merged into one code repository creation task, because when a new code repository is indexed, the latest version of the code will be pulled for indexing, so the newly submitted content will be included. If there are multiple code update tasks and it contains a repository archiving or deletion task, they will be merged into one repository deletion task to avoid Code Search index resource consumption.
[0048] This step can significantly reduce the number of repeated and invalid tasks through the above queue optimization logic, reduce the invalid loss of Code Search indexing, and speed up the index building time of the latest version of the warehouse.
[0049] In this embodiment, the search service will poll the task queue regularly according to the preset period and obtain the tasks in the queue. After the task is obtained, the search service will parse the parameters in the task and perform the index update operation according to the configured concurrency number. This mechanism supports not only the concurrent index construction of a single shard, but also the parallel index construction of multiple shards.
[0050] When there are multiple search service shards, each shard will periodically and in parallel obtain the code base update tasks related to itself from the index building queue and build the index within the shard. This architectural design can significantly improve the efficiency of index building when there are a large number of code bases and a high number of concurrent updates.
[0051] When the single-shard task load is heavy, the warehouse of the high-frequency shard can be scheduled to the idle shard to further optimize resource allocation, avoid the performance bottleneck of a single shard, and ensure the efficient completion of the index update and construction tasks. Through the multi-shard parallel processing mechanism of this architecture, it can better cope with the high-frequency code library update requirements and search requests on large-scale code platforms, ensure the real-time index update and the efficiency of search results, and enable the system to maintain stable and efficient performance in large data volume scenarios.
[0052] When the search service builds a code index based on the index building task, it specifically includes:
[0053] Step a: Index initialization: The search service will pull the full repository code of the default branch and the active branch to the disk of the search service. After the pull is completed, a complete index file will be built for the directory and stored on the disk for code search requests;
[0054] Step b: Create a new warehouse: Same as the index initialization step, after the warehouse is created, the full index file of the warehouse will be created, and the source code of the warehouse will be stored in the disk of the search service as cache data for incremental construction;
[0055] Step c: Incremental update: The search service will first find the repository that needs to be updated, incrementally pull the latest version of the repository to the disk for code search, and then update the index file based on the incremental content to avoid resource loss caused by pulling the full code and building the index, thereby speeding up the code index building process.
[0056] Step d: Index deletion: The search service will delete the index files that have been built in the code cache of the code library on disk based on the specific deletion parameters to prevent search resources from being occupied by abandoned repositories.
[0057] In order to speed up the index construction, this embodiment supports setting the branches that the warehouse needs to index to avoid that some infrequently used branch updates will also occupy index resources. At the same time, it can more accurately index the branches that users really need, thereby speeding up the efficiency of index construction.
[0058] In this embodiment, after the index building fails, the search service provides a retry mechanism. The process is as follows: Figure 3 As shown. Currently, the default supports 5 retries, and the number of retries can be manually adjusted. The first retry will execute the general index building steps again, first pull the incremental update code, and then execute the index building steps. If it fails again, it will try to delete the cached code directory and re-clone the code of this repository in full to ensure that the code is downloaded successfully. After the code is pulled successfully, the index building task will be triggered again. If the index incremental update task fails, it will try to delete the index file and re-build the index in full to ensure that the index task is successfully built.
[0059] After the task is processed, the Code Search service will call the callback interface of the index queue to notify that the task has been completed. If the index task fails, the reason for the failure will be transmitted back and recorded in the queue to facilitate subsequent maintenance personnel to troubleshoot the cause.
[0060] This embodiment provides a method for quickly building a code search index, which can quickly complete incremental index updates after the user submits the code, so that the newly submitted code content can be quickly searched. This method can not only improve the speed of index updates, but also ensure good scalability when facing large-scale code bases, so as to better meet the needs of developers under high-frequency iterations.
[0061] The method for quickly building a code search index provided in this embodiment efficiently manages index updates by maintaining a code index synchronization queue. When a code update occurs in a repository in the code platform, the synchronization task is immediately added to the synchronization queue and consumed by the code index maintenance service. In order to avoid repeated processing of the same update task, the queue also has the ability to aggregate messages, which can prevent repeated consumption and thus save system resources.
[0062] After receiving the synchronization task, the consumer side will incrementally pull the updated code content and generate indexes only for these incremental parts. This method greatly improves the speed of index updates and reduces the time and resources required for full updates. In addition, the architecture design of the search service supports multi-shard deployment and can process warehouse code updates from different nodes in parallel. This distributed parallel processing mechanism can significantly reduce the waiting time for index updates in concurrent scenarios and improve the efficiency and performance of the overall index update processing.
[0063] The method for quickly building a code search index provided in this embodiment also supports the configuration of warehouse index branches, allowing users to conveniently manage specific branch content that needs to be indexed, thereby reducing the time consumption of establishing full index branches. Through this branch index configuration, users can flexibly select branches that need to be concerned about, avoiding unnecessary waste of resources and time delays.
[0064] This embodiment can effectively cope with frequent code updates and large-scale code bases while ensuring the real-time nature of the index and the scalability of the system. This embodiment also further optimizes its scalability to enable it to maintain stable and efficient performance when facing large-scale code bases. Through multi-shard deployment and parallel processing mechanisms, it can be dynamically expanded according to the load conditions of different warehouses to support horizontal expansion capabilities. As the size of the code warehouse grows or the frequency of updates increases, new nodes or shards can be easily added to balance the load and ensure that response time is not affected.
[0065] This embodiment solves the problems of delayed index updates, repeated processing, and insufficient concurrent processing capabilities in existing code search systems, allowing developers to quickly perform search operations after code submission, greatly improving development efficiency. At the same time, it can maintain high performance when processing a large number of concurrent tasks, providing technical support for the management and retrieval of large-scale code repositories.
[0066] In summary, this embodiment has the following advantages:
[0067] (1) The efficiency and flexibility of code retrieval are greatly improved through real-time index updates and multi-shard expansion capabilities.
[0068] (2) In terms of index updates, a real-time and incremental update mechanism is adopted to ensure that changes to the code base can be quickly synchronized to the index.
[0069] (3) Through multi-shard parallel construction and branch management capabilities, the speed of index updates is further improved. In terms of multi-shard expansion, the shard storage structure and dynamic expansion capabilities support the processing of large-scale code bases and high-concurrency requests. Load balancing and fault tolerance mechanisms ensure stable operation under high load or failure, ensuring efficient performance and stability under high concurrency and large-scale data conditions. According to test results, the code repository can complete the establishment of code indexes in seconds at the fastest after being updated.
[0070] Embodiment 2
[0071] This embodiment provides a device for quickly building a code search index, including:
[0072] An index task request receiving module, used to receive an index task request sent by a search service;
[0073] A registration module, used to register the unregistered Code Search shard as a valid search shard according to the index task request;
[0074] An initialization module is used to perform a modulus operation based on the warehouse ID and the number of valid search shards, and record the association between the warehouse and the shards to obtain an index building queue;
[0075] The index building queue optimization module is used to update the order of the index building queue according to a preset priority, and merge the index building tasks of the same warehouse according to a preset task merging strategy to obtain an optimized index building queue; the search service regularly polls the optimized index building queue according to a preset period, obtains the index building tasks, and builds the code index according to the index building tasks.
[0076] For the specific implementation content of each module in a device for quickly building a code search index, please refer to the definition of a method for quickly building a code search index above, which will not be repeated here.
[0077] The technical features of the above embodiments may be arbitrarily combined (as long as there is no contradiction in the combination of these technical features). In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described; these embodiments that are not explicitly written should also be considered to be within the scope of this specification.
Claims
1. A method for quickly constructing a code search index, characterized in that: The method is applied to the index queue service, and includes: Step 1: Receive the indexing task request sent by the search service; Step 2: Register the unregistered Code Search shard as a valid search shard according to the index task request; Step 3: Perform a modulus operation based on the warehouse ID and the number of valid search shards, and record the association between the warehouse and the shards to obtain an index building queue; Step 4: Update the order of the index building queue according to the preset priority, and merge the index building tasks of the same warehouse according to the preset task merging strategy to obtain the optimized index building queue; the search service regularly polls the optimized index building queue according to the preset period, obtains the index building tasks, and builds the code index according to the index building tasks; the search service supports concurrent index building of a single shard and parallel index building of multiple shards; The search service builds the code index according to the index building task, which specifically includes: index initialization, new repository, incremental update and index deletion; If the search service fails to build the code index according to the index building task, it will first pull the incremental updated code, and then execute the index building steps; if it fails again, it will try to delete the cached code directory and re-clone the code of this repository in full to ensure successful code download. After the code is pulled successfully, the index building task will be triggered again. If the index incremental update task fails, it will try to delete the index file and re-build the index in full to ensure successful indexing task construction.
2. The method for quickly constructing a code search index according to claim 1, characterized in that: In step 1, the index task request includes: the address of the search service itself, the shard name and the disk information.
3. The method for quickly constructing a code search index according to claim 1, characterized in that: In step 4, when the order of the index building queue is updated according to the preset priority, specifically: warehouses with large priority weights are placed at the front of the index building queue, and warehouses with consistent weights are sorted according to code update time.
4. The method for quickly constructing a code search index according to claim 1, characterized in that: In step 4, the index building tasks of the same repository are merged according to the pre-set task merging strategy: if there are multiple code update tasks, they are merged into one code update task; if there is a new repository task and it contains multiple code update tasks, they are merged into one code repository new task; if there are multiple code update tasks and it contains a repository archiving or deletion task, they are merged into one repository deletion task.
5. The method for quickly constructing a code search index according to claim 1, characterized in that: When the index is initialized and a new warehouse is created, a complete index file will be constructed and stored in the disk of the search service.
6. The method for quickly constructing a code search index according to claim 1, characterized in that: The incremental update specifically includes: obtaining the repository that needs to be updated, incrementally pulling the latest version of the repository to the disk for code search, and updating the index file based on the incremental content.
7. A device for quickly building a code search index, characterized in that: include: An index task request receiving module, used to receive an index task request sent by a search service; A registration module, used to register the unregistered Code Search shard as a valid search shard according to the index task request; An initialization module is used to perform a modulus operation based on the warehouse ID and the number of valid search shards, and record the association between the warehouse and the shards to obtain an index building queue; An index building queue optimization module is used to update the order of the index building queue according to a preset priority, and merge the index building tasks of the same warehouse according to a preset task merging strategy to obtain an optimized index building queue; the search service regularly polls the optimized index building queue according to a preset period, obtains the index building tasks, and builds a code index according to the index building tasks; The search service supports concurrent index building of a single shard and parallel index building of multiple shards; The search service builds the code index according to the index building task, which specifically includes: index initialization, new repository, incremental update and index deletion; If the search service fails to build the code index according to the index building task, it will first pull the incremental updated code, and then execute the index building steps; if it fails again, it will try to delete the cached code directory and re-clone the code of this repository in full to ensure successful code download. After the code is pulled successfully, the index building task will be triggered again. If the index incremental update task fails, it will try to delete the index file and re-build the index in full to ensure successful indexing task construction.