Multi-source retrieval unique route registration and capacity chunk fanout method and system

Abstract

The application discloses a multi-source retrieval unique routing registration and capacity block fan-out method and system. The method comprises the following steps: maintaining a source registration table, registering each necessary source and its routed sound part, applicable level and optional grouping attribute; checking the unique routing invariability during the registration table loading period, and determining the import failure when there is repeated routing; responding to the query, filtering and excluding the non-connected person according to the registration table mode and level, and determining the source set to be searched this time; for the sound part retrieved by the site limited operator, grouping the source to be searched according to the category or theme, sorting according to the registration table registration order, and cutting into several fan-out blocks by the upper limit of the single query operator; each fan-out block is used as a query to initiate retrieval, the block results are merged into the same retrieval result set, the resource locator is removed, and the source coverage check is performed on the returned result. The application can correctly merge the cross-block results, and the same input can obtain the same block plan, and the retrieval coverage is reproducible.

Description

Technical Field

[0001] This invention relates to the field of information retrieval technology, specifically to a unique route registration and capacity block fan-out method and system for multi-source retrieval. Background Technology

[0002] In multi-source retrieval systems, it is often necessary to distribute the same query request to multiple pre-registered mandatory sources for retrieval. Some practices already exist that combine the sites in the list into a single retrieval query and send it out at once, or send out large sets in batches according to a fixed number.

[0003] In the field of distributed data retrieval, patent document US10476662B2 discloses a method for operating a distributed key-value store, which partitions data records by identifiers and builds encrypted indexes on the client side to support distributed queries. This approach distributes query requests to multiple server nodes for execution, but does not uniquely assign ownership of each source among multiple retrieval channels, nor does it verify the consistency of the query parameters actually carried in each batch with the predetermined distribution plan. Regarding batch retrieval orchestration, existing systems often divide large collections into fixed batches, but the batching granularity does not correspond to the number of site-specific operators that a single query by the retrieval engine can handle, causing the engine to silently discard some site-specific operators without returning an error when the limit is exceeded.

[0004] The above approach has the following drawbacks: the source route is not unique, which leads to the same site being retrieved by multiple voices, repeatedly consuming the quota; the granularity of the block is decoupled from the engine's carrying capacity, and the site is silently dropped after the limit is exceeded; there is a lack of return result coverage verification, and the system cannot detect the site after the engine silently drops it; the block plan of the same query is not reproducible because it depends on dynamic sorting. Summary of the Invention

[0005] This invention modifies the internal retrieval task scheduling and execution process of a computer system: import-time verification and runtime dispatch constraints of the source registry, deterministic block partitioning based on the upper limit of the single query capability of the retrieval engine, and source coverage verification of the returned results. The technical problem solved is how to ensure the retrieval coverage of essential sources is deterministic and reproducible under the constraint that a given retrieval engine has an upper limit on single query operators and silently degrades when the limit is exceeded. The achieved technical effect is that each essential source falls exactly into a search block that does not exceed the limit and is dispatched without being searched repeatedly, while the situation where the engine silently discards a site can be detected and located.

[0006] The technical problems to be solved by this invention include: how to ensure that each mandatory source is indeed dispatched for retrieval and is not repeatedly retrieved, given the limit on the number of single query operators; how to transform the silent discarding of over-limit queries into deterministic coverage; how to make the identity of each block identifiable and the results correctly merged when the same group spans multiple blocks; and how to make the retrieval coverage reproducible.

[0007] This invention provides a method for unique route registration and capacity block fan-out for multi-source retrieval, comprising: Maintain a source registry, registering each mandatory source and its routed voice, applicable level, retrieval availability status, and optional category attributes and topic labels for grouping; the same mandatory source may have one or more registration records in the registry; Before responding to any query, the method verifies the unique route invariant during the registry loading period: it traverses all mandatory sources in the registry, and if a duplicate route exists, it determines that the import has failed, outputs the source identifier of the duplicated route and its component identifiers, and prevents the method from entering a query-responsive state; the following steps are executed only after the verification passes. In response to a query, the query mode parameters carried in the query request are read, and the system filters according to the query mode and applicable level based on the registry, excluding those whose available status is "not connected," to determine the set of sources to be queried this time. Each source in the set of sources to be queried this time is retrieved only by the voice department registered in the registry, and not by other voice departments. The same source does not appear in the current dispatch plan of two or more voice departments. For a voice segment retrieved using site-qualified operators, the sources routed to that voice segment are grouped according to their category or topic. Those carrying category attributes are grouped according to the value of their category attributes; those carrying topic tags but not category attributes are grouped according to the first value in their topic tags; those carrying neither category nor topic tags are grouped into a single default group; each source is grouped into only one group; after sorting each group according to the registration order of each source in the registry, it is divided into several fan-out blocks by the upper limit of the number of site-qualified operators allowed to be carried in a single query, which is pre-configured for that voice segment. Each source corresponds to one site-qualified operator. Based on the number of sources, each source is assigned to exactly one fan-out block, and the number of sources in each block does not exceed the upper limit. Each block carries a stable block identifier consisting of a group, a block number, and an ordered list of sources. Each fan-out block is retrieved as a query sent to the voice part. The results of each block are merged into the same search result set and deduplicated according to the resource locator. For each fan-out block, the sources appearing in the returned results of the block are compared with the ordered source list carried by the stable block identifier of the fan-out block. If there are sources listed in the ordered source list but not appearing in the returned results, the fan-out block is marked as incomplete coverage and its stable block identifier and the missing source identifier are output. For sources registered to voice parts not subject to the upper limit in the current source set to be searched, no segmentation is performed. Instead, a retrieval request is initiated directly for that voice part, and the results are merged into the same search result set.

[0008] Preferably, the loading period verification unique routing invariant includes: traversing all mandatory sources in the registry and verifying the voice field to which each source is routed; when any source is repeatedly routed by two or more voices, it is determined as an import failure, and the voice identifier of the source that is repeatedly routed is indicated; the import failure prevents the system from entering an acceptable query running state.

[0009] Preferably, after determining the set of sources to be searched this time and before the grouping and segmentation, the method further includes topic gating: probing the topics involved in the current query based on the content of the current query, retaining sources with topic labels only if their topics are relevant to the current query, and always retaining sources without topic labels; after topic gating, sources that are irrelevant to the current query are pruned from the set of sources to be searched this time, so as to save retrieval quota.

[0010] Preferably, each source entry in the registry is constructed by verifying the validity of its field values: the domain name field is verified to ensure its format conforms to a predetermined specification; the routed voice, applicable level, and retrieved availability fields are verified to ensure their values ​​belong to their respective predetermined enumeration sets; and the optional category attributes are verified to ensure their values ​​belong to a predetermined category enumeration set. If any field fails verification, an error is thrown during construction, and the creation of the entry is rejected. After the source entry is constructed, it is stored in the read-only area of ​​the registry. The registry access interface provided to the outside world during runtime only exposes the read method and does not provide any write, update, or delete methods for any field of the entry. Write requests for any field of the entry are rejected and recorded in the audit log.

[0011] Preferably, the step of dividing the group into several fan-out blocks based on the upper limit of the number of site-limited operators allowed to be carried in a single query, which is pre-configured for the voice part, includes: for each group, after stable sorting according to the registration order of each source in the registry, counting sequentially from the first source of the group, and dividing it into a block when the upper limit is reached, so that the number of sources contained in each block does not exceed the upper limit; if the upper limit is less than one, an error is thrown and no unusable empty block plan is generated; each source corresponds to one site-limited operator, counted according to the number of sources.

[0012] Preferably, the stable block identifier includes: the group identifier to which the fan-out block belongs, the block number within the group starting from zero, and an ordered list of sources contained in the block arranged in the registration order; the stable block identifier ensures that when the same group spans multiple blocks due to the number of sources exceeding the upper limit, each block still retains its identifiable identity, and the cross-block results are correctly merged.

[0013] Preferably, the process of initiating retrieval for each fan-out block as a query sent to the sound part includes: instantiating the corresponding retrieval executor for each fan-out block and initiating retrieval, with each block retrieval being executed in parallel; the process of merging the results of each block into the same search result set and removing duplicates by resource locator includes: merging the search results of all fan-out blocks into the same search result set and removing duplicate hits by resource locator; marking retrieval as complete only after all fan-out blocks are successfully retrieved; and explicitly returning an incomplete status if any fan-out block retrieval fails, without presenting partial results as complete.

[0014] Preferably, the grouping, the intra-group sorting, and the segmentation are input only to the registration order of each source in the registry, the grouping attributes of each source, and the pre-configured upper limit; the registry is stored in an order-preserving structure and traversed according to the registration order, and the intra-group sorting is a stable sort; thereby ensuring that when the registry content, the query mode, and the query are the same, the generated fan-out block list and its stable block identifier are identical one by one.

[0015] Preferably, the voice part includes voice parts retrieved using the site-limited operator method, voice parts retrieved using the allowed domain list method, and voice parts retrieved using the broad search method; for voice parts that are not subject to the upper limit constraint of the site-limited operator, their corresponding search sources are not divided into blocks, and the retrieval executor is directly instantiated to initiate retrieval.

[0016] This invention also provides a unique route registration method for multi-source retrieval, comprising: Maintain a source registry, registering each mandatory source, its routed voice part, applicable level, and retrieval availability status. During the registry loading period, the unique route is verified as unchanged: if a duplicate route exists, the import is deemed to have failed, and the method is not allowed to enter a query-responsive state; the following steps are executed only after the verification passes. In response to a query, the query mode parameters are read, and the system filters according to the query mode and applicable level based on the registry, excluding those whose available status is "not connected," to determine the set of sources to be queried this time. Each source in the set of sources to be queried this time is retrieved only by the voice department registered in the registry, and not by other voice departments. The same source does not appear in the current dispatch plan of two or more voice departments. The results of retrieval initiated separately by each voice part are merged into the same search result set and deduplicated according to the resource locator.

[0017] Preferably, each source entry in the registry is constructed by verifying the validity of its field values: the domain name field is verified to ensure its format conforms to a predetermined specification; the routed voice, applicable level, and retrieved availability fields are verified to ensure their values ​​belong to their respective predetermined enumeration sets; if any field fails verification, an error is thrown during construction and the creation of the entry is rejected; after the source entry is constructed, it is stored in the read-only area of ​​the registry. The registry access interface provided to the outside world during runtime only exposes the read method and does not provide any write, update, or delete methods for any field of the entry. Write requests for any field of the entry are rejected and recorded in the audit log.

[0018] This invention also provides a unique route registration and capacity block fan-out system for multi-source retrieval, comprising: The registry maintenance module is used to maintain a source registry, registering each mandatory source and its routed voice, applicable level, retrieval availability status, and optional category attributes and topic labels for grouping. The import period verification module is connected to the registry maintenance module and is used to verify the unique routing invariant during the registry import period, and to prevent the system from entering the service state that can receive queries when the verification fails. The source determination module is connected to the registry maintenance module. In response to a query, it filters the data according to the registry based on the query mode and applicable level, and excludes those whose available status is not connected, to determine the set of sources to be queried this time. The deterministic grouping and segmentation module, connected to the source determination module, is used to, for a voice segment retrieved using site-limited operators, group those sources in the current source set that are routed to that voice segment into one or more groups according to their category or topic. Each source is grouped into only one group, and those without grouping attributes are grouped into a single default group. After sorting each group according to the registration order of each source in the registry, it is divided into several fan-out blocks based on the upper limit of the number of site-limited operators allowed to be carried in a single query, which is pre-configured for that voice segment. Each source corresponds to one site-limited operator. Based on the number of sources, each source that is routed to that voice segment is assigned to exactly one fan-out block, and the number of sources in each block does not exceed the upper limit. Each block carries a stable block identifier consisting of a group, a block number, and an ordered list of sources. The regression module, connected to the deterministic grouping and segmentation module, is used to retrieve each fan-out block as a query sent to the sound part, merge the results of each block into the same retrieval result set and remove duplicates according to the resource locator. The source coverage verification module, connected to the regression and union module, is used to compare the sources appearing in the returned results of each sector with the ordered source list contained in the stable block identifier of the sector. If there are sources listed in the ordered source list but not appearing in the returned results, the sector is marked as incompletely covered and its stable block identifier and the missing source identifier are output. The wide-search sound part processing module, connected to the source determination module, is used to, for sound parts not subject to the upper limit constraint, directly instantiate the retrieval executor to initiate retrieval without performing the segmentation, and merge the results into the same retrieval result set.

[0019] The present invention also provides an electronic device, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a unique route registration and capacity block fan-out method for multi-source retrieval.

[0020] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described method for unique route registration and capacity block fan-out for multi-source retrieval.

[0021] The present invention also provides a computer program product, including computer program code or instructions, which, when executed on a computer, enable the implementation of a unique route registration and capacity block fan-out method for multi-source retrieval.

[0022] The beneficial effects of this invention include: This invention verifies the unique route invariant during the registry import period, ensuring that each mandatory source is routed by exactly one voice, reducing duplicate searches and quota consumption. Furthermore, it advances the exposure of duplicate / ambiguous routes from the runtime to the import period, terminating the loading process via a loading failure termination method instead of silently continuing.

[0023] This invention uses deterministic exhaustive block division based on the upper limit of a single query operator to ensure that each mandatory query source falls into a block that does not exceed the limit for retrieval, transforming the silent discarding of exceeding the limit into deterministic dispatch and reducing coverage drift.

[0024] This invention enables the detection and location of situations where the engine silently discards sites by performing source coverage verification on the returned results, thus avoiding the output of incomplete search result sets as complete results to downstream applications.

[0025] This invention uses stable block identification and deterministic block partitioning plan to ensure that the same input will always yield the same block partitioning plan, and the retrieval coverage can be reproduced. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of the overall process of the unique route registration and capacity block fan-out method for multi-source retrieval provided in an embodiment of the present invention; Figure 2 A schematic diagram illustrating the process of unique route verification during the import period of the source registry, provided for an embodiment of the present invention; Figure 3 A flowchart illustrating the process of determining the source set to be investigated and subject gating in this embodiment of the invention; Figure 4 A flowchart illustrating deterministic grouping and capacity partitioning provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the process of retrieving and merging duplicates for each block provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the architecture of a unique route registration and capacity block fan-out system for multi-source retrieval provided in an embodiment of the present invention; Figure 7 This is a hardware structure diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0029] The relevant terms in this invention are defined as follows: A voice unit refers to an execution unit in a retrieval system that has independent retrieval channels and retrieval capabilities, including channels that retrieve data using site-limited operators, channels that retrieve data using allowed domain lists, and channels that retrieve data using broad search.

[0030] "Required sources" refers to a collection of sites or resources that are pre-registered in the source registry and must be covered in the search.

[0031] Site qualifying operators are query syntax elements that limit the search scope to a specific site or domain.

[0032] A registry snapshot is a read-only copy of the state of all entries in the source registry at a certain loading completion point, which serves as the basis for calculating a single query block plan.

[0033] The retrieval actuator is a software module responsible for initiating retrieval requests to a specific voice and receiving the returned results.

[0034] A block plan is a set of fan-out blocks determined by grouping rules, registration order, and a predetermined upper limit. Under the same snapshot, the same query will always get the same plan.

[0035] Resource identifiers are data tags used to identify the source of search results, including but not limited to Uniform Resource Locators or Content Addressing Hashes.

[0036] Load failure termination refers to the system's handling method when the registry loading verification fails, in which case the system determines the error status and terminates the loading process, without entering a responsive running state.

[0037] like Figure 1 As shown, this embodiment provides a unique route registration and capacity block fan-out method for multi-source retrieval. The method is divided into four sequentially connected processing stages, corresponding to source registry maintenance and import period verification, determination of the set of sources to be queried, deterministic grouping and block partitioning, and retrieval and deduplication of each block. Each stage is coupled to the same data integrity link.

[0038] like Figure 2As shown, during the import phase, the system first maintains a source registry, registering each mandatory source entry one by one. The registry is stored in a structured data format, with each mandatory source corresponding to one registration record.

[0039] In one implementation, the registration record includes the following fields: Domain: The domain identifier of the mandatory source, serving as a unique identifier for the source; Voice: The identifier of the retrieval voice to which the source is assigned, taken from a predefined voice enumeration set; Level: The retrieval level to which the source applies, taken from a predefined level enumeration set; Availability: Indicates whether the source can currently be retrieved through its assigned voice, including predetermined enumeration values ​​such as "connected", "not connected", and "restricted"; Category: An optional field used for grouping, taken from a predefined category enumeration set; Topic: An optional field used for topic gating, in the form of a string or a list of strings.

[0040] Each source entry undergoes a validation process during construction, verifying the validity of the values ​​in each of the aforementioned fields. This validation includes: whether the field value is non-empty, whether the field value belongs to its corresponding predefined enumeration set, and whether the domain name format conforms to predefined specifications. If any field validation fails, an error is thrown during construction, rejecting the creation of the entry, rather than silently accepting erroneous data.

[0041] Specifically, after the source entry passes the construction verification, it is stored in the read-only area of ​​the registry. The registry access interface provided during runtime only exposes the read method and does not provide methods for writing, updating, or deleting any field of the entry. Write requests for any field of the entry are rejected and recorded in the audit log. When it is necessary to change the routing relationship or other fields of a certain source, it is done by reloading the registry. During reloading, the aforementioned construction-time field verification and import-time unique routing invariant verification are re-executed for all entries.

[0042] After all entries in the registry are registered, the system enters the unique route verification phase during the import period. The system traverses all mandatory sources in the registry, extracts the "routed voice part" field from each record, and establishes a mapping table from source identifiers to voice part identifiers. Subsequently, the system checks if any source identifier in this mapping table corresponds to two or more different voice part identifiers. If so, it is determined that the source has been routed repeatedly, and the system immediately terminates the loading process with an import failure termination method, outputting import failure information. This import failure information includes the repeatedly routed source identifier and the corresponding voice part identifiers, allowing maintenance personnel to locate configuration errors. The import failure prevents the system from entering an acceptable query running state, ensuring that duplicate searches or missed checks do not occur during runtime due to routing ambiguity. Each reload or incremental registration of the registry re-executes the unique route invariant verification.

[0043] In one implementation, the import period verification also includes verifying the validity of the voice part identifier: checking whether the voice part identifier in each record exists in the list of voice parts currently registered in the system. If an unregistered voice part exists, the import is also determined to have failed and loading is terminated.

[0044] like Figure 3 As shown, during runtime, in response to a query request, the system first determines the set of sources to be queried based on the registry. This determination process includes the following sub-steps: First, pattern filtering: The system reads the query pattern parameter carried in the query request and filters out the required sources from the registry whose applicable level matches the query pattern. For example, when the query pattern is "deep search", only source entries whose applicable level contains "deep" or "all" are retained.

[0045] Second, status filtering: The system excludes source entries with a status of "not connected" from the retrieved results. For source entries with a status of "restricted", the system retains them but marks them as restricted. This mark is returned with the search results and is not used in the grouping, segmentation, and deduplication judgment.

[0046] Third, topic gating (optional): If topic gating is enabled in the system configuration, topic relevance pruning is performed on the source set after the above filtering. The system probes the topics involved in the current query: for Latin letter keywords, word boundary matching is used to avoid substring false hits; for Chinese, Japanese, Korean, and other characters that are not segmented by whitespace, substring matching is used. After the probe is completed, the system obtains the topic set for the current query. Subsequently, the system traverses each source in the source set: if the source has no topic label, it is always retained; if the source has a topic label, it is only retained if its topic label intersects with the topic set of the current query, otherwise it is pruned from the source set to be queried. After the topic gating, sources unrelated to the current query will no longer occupy subsequent block quotas and retrieval quotas.

[0047] In one implementation, the topic detection employs a two-level matching strategy: the first level is exact matching, detecting whether the topic keywords appear completely in the query content; the second level is word boundary matching (for Latin letters) or substring matching (for Chinese, Japanese, and Korean characters) to capture variant forms. The topic keyword table is pre-configured by operations and maintenance personnel and maintained in groups according to source type.

[0048] In one implementation, after the set of sources to be queried is determined and before grouping and partitioning are performed, the system divides the set into different voices according to the routing relationships registered in the registry: each source to be queried only enters the current dispatch plan of the voice it is registered in the registry, and does not enter the dispatch plans of any other voice; the same source does not appear in the current dispatch plans of two or more voices at the same time. Thus, the unique routing invariant verified during the import phase is actually applied to each query in the form of a dispatch plan during runtime, rather than merely remaining a structural constraint of the registry itself.

[0049] like Figure 4 As shown, for a voice part retrieved using the site-limited operator, the system performs deterministic grouping and chunking of the sources routed to that voice part in the current source set to be searched. The process includes the following sub-steps: First, grouping: The system retrieves all sources routed to the specified voice part and categorizes them into one or more groups based on their category or topic attributes. Specifically, if a source carries a category field, it is grouped by category value; if a source carries a topic field but not a category field, it is grouped by topic value; if a source carries neither a category nor a topic, it is placed into a single default group. These grouping rules ensure that each source is assigned to only one group, preventing duplicates across groups.

[0050] In one implementation, when the source's topic tags are in the form of a string list and the source does not carry a category field, the first value in the list is taken as the grouping key for the source, and the remaining values ​​in the list are not included in the grouping. The order of the values ​​in the list is the original order in which the source was registered in the registry, without any rearrangement. Thus, sources with multiple topic tags are still grouped into only one group, and the constraint of "each source is grouped into only one group" also holds when the topic tags are a list, and the same registry content always yields the same grouping result.

[0051] Second, intra-group sorting: For sources within each group, a stable sorting is performed according to the order in which each source is registered in the registry. This registration order is the physical order in which each entry is registered in the registry, independent of lexicographical order, hash order, or random order. This stable sorting ensures that the same input yields the same intra-group order at any time and on any machine, providing a foundation for the determinism of the partitioning plan.

[0052] Third, capacity partitioning: For each sorted group, the system uses the pre-configured upper limit for the number of site-limited operators allowed in a single query for that voice section as a boundary. Starting from the first source of the group, the count is incremented sequentially. Each time the upper limit is reached, the group is partitioned into a block. This partitioning ensures that each source fits into a block, and the number of sources in each block does not exceed the upper limit. If the upper limit is not reached after the last source of the group is counted, the last source and the previously accumulated sources together form the last block. If the upper limit is configured to a value less than one, the system immediately throws an error and does not generate an unusable empty block plan.

[0053] Fourth, stable block identifier generation: After each block is generated, the system assigns a stable block identifier to the block. In one implementation, the stable block identifier is a triple structure, including: the group identifier (such as category name or topic name) to which the fan-out block belongs, the block number starting from zero within the group, and an ordered list of sources contained in the block arranged in the order of registration. For example, a group "finance" contains 35 sources and has a predetermined upper limit of 10, and is divided into 4 blocks, with stable block identifiers of ("finance", 0, [src_001, ..., src_010]), ("finance", 1, [src_011, ..., src_020]), ("finance", 2, [src_021, ..., src_030]), and ("finance", 3, [src_031, ..., src_035]). The stable block identifier ensures that when the same group spans multiple blocks due to the number of sources exceeding the upper limit, each block retains its identifiable identity, and subsequent cross-block results can be correctly merged.

[0054] In one implementation, for voices that are not subject to the upper limit of site-limited operators (such as voices retrieved using an allowed domain list or a broad search), their corresponding sources are not divided into blocks, and the retrieval executor is directly instantiated to initiate retrieval.

[0055] like Figure 5 As shown, each sector block, as a query sent to the corresponding voice unit, instantiates a retrieval executor and initiates a retrieval. The retrieval executor, based on the predefined query expression of that voice unit, assembles the site-specific operators from each source within the block into a complete retrieval query. The retrieval executors of each sector block can execute in parallel to shorten the overall retrieval time.

[0056] After each sector block is retrieved, the system merges the results of each block into the same search result set. The merging process includes the following sub-steps: First, result aggregation: The search results returned by each block are sequentially aggregated into the same result container according to the stable block identifier order of each sector.

[0057] Second, deduplication: The aggregated search result set is deduplicated according to the resource locator. If the same resource is hit repeatedly by multiple fan-out blocks, only the first occurrence is retained, and subsequent duplicate records are discarded.

[0058] Third, source coverage verification: For each outgoing block, the system extracts the source of each result from the search results returned for that block and compares it item by item with the ordered source list contained in the stable block identifier of that outgoing block. If there is a source listed in the ordered source list but not present in any of the returned results, the outgoing block is marked as having incomplete coverage, and the stable block identifier of that outgoing block and the identifier of the missing source are output. When the number of site constraint operators carried by the search engine exceeds its actual carrying capacity, it usually does not return an error, but silently discards some of the site constraints. The discarded sites therefore do not participate in this search and do not generate any prompts. The above-mentioned source coverage verification of the returned results allows this situation to be detected and located to the specific outgoing block and the specific source, so as not to output an incomplete search result set as a complete result downstream.

[0059] Fourth, status determination: The system checks the retrieval status of all fan-out blocks. If all fan-out blocks are retrieved successfully, the retrieval is marked as complete, and a complete retrieval result set is returned. If any fan-out block retrieval fails (including timeout, connection exception, return error code, etc.), the system explicitly returns an incomplete status, and includes the identifier of the failed fan-out block and the reason for the failure in the response, without presenting partial results as complete results.

[0060] In one implementation, the retrieval actuator also performs quota management functions: each time a retrieval is initiated, a consumption is registered with the quota counter; if the quota is exhausted, the retrieval of subsequent fan-out blocks is suspended and the quota shortage status is returned.

[0061] The same query, the same query pattern, and the same registry snapshot will always yield the same block plan. This determinism is guaranteed by the following factors: Determinism of the source set: uniquely determined by registry content, query pattern, and applicable hierarchical filtering rules; Determinism of the grouping rules: category or topic attributes are predefined fields in the registry, and the grouping logic is unambiguous; Determinism of the sorting rules: strictly based on the registry registration order, without introducing random or dynamic factors; Determinism of the partitioning rules: a predetermined upper limit is a configuration constant, and the partitioning algorithm is a simple sequential cumulative counting. Therefore, for the same input conditions, regardless of when or on which machine the system executes, the generated fan-out block list is identical, and the retrieval coverage is reproducible.

[0062] The following synergistic effects exist between the features of this invention, indicating that this solution is not a simple superposition of known methods: (i) Coupling between the upper limit of a single query operator and the granularity of the blocks: The size of each block is exactly bounded by the "upper limit of a single query operator", ensuring that each block will not exceed the limit. It is this coupling that transforms "silently discarding when the limit is exceeded" into "definite overwriting". If the granularity of the blocks is decoupled from this upper limit, it may still exceed the limit or be over-segmented.

[0063] (ii) The coupling of the unique route invariant and concurrent merging deduplication during the import period: the unique route eliminates duplicate retrieval at the source, and the concurrent merging deduplication is carried out by resource identifier. The coupling of the two makes the "each source is retrieved exactly once" true before dispatch and verifiable after aggregation.

[0064] (iii) Coupling of subject gating and deterministic block partitioning: gating first cuts out the relevant sources for this time, and then the partitioning is further divided into blocks after the cut groups, so that the quota is only spent on the relevant sources, and the partitioning plan is determined according to (mode, query).

[0065] (iv) Stable block identification and cross-block merging coupling: Stable identification enables multiple blocks in the same group to be identified and merged into one. Without stable identification, cross-block results cannot be merged correctly.

[0066] (v) Returning results source coverage verification and chunking plan coupling: The verification is based on the ordered source list in the chunking plan to detect the engine's silent discard of specific sites. If there is no chunking plan, it is impossible to know which sources should be covered by a certain block.

[0067] Removing any one of the couplings results in the other method degenerating—therefore, compared to the simple superposition of "separately known site retrieval, query merging, and batching of sets," it has a non-obvious overall technical effect.

[0068] In one improvement process, the early design did not define the relationship between block size and the upper limit of a single query; the block size was determined by a fixed number (e.g., 10 sources per block). It was later discovered that the upper limit of a single query varied across different voices, and the fixed block size still exceeded the limit for some voices, resulting in silent discarding, while for others it led to excessive fragmentation and increased query counts. Therefore, the design was changed to use the upper limit of a single query operator for each voice as the block granularity, ensuring that each block would not exceed the limit.

[0069] The early design did not verify route uniqueness during the import phase. Duplicate routes were only exposed as duplicates when the source was happened to be retrieved during runtime, resulting in a long latency period and difficulty in locating the problem. The design was later changed to verify unique route invariants during the import phase by terminating the process upon loading failure, thus exposing configuration errors at system startup.

[0070] Early designs did not carry stable identifiers for blocks, only assigning unstructured serial numbers. When the same group spanned multiple blocks, the merging phase could not determine which blocks belonged to the same group, leading to difficulties in merging cross-block results, duplicate sources, or omissions. Later, a stable block identifier consisting of the group, block number, and ordered source list was added, making cross-block identities identifiable and ensuring correct result merging.

[0071] The early design did not perform source coverage verification on the returned results. Even after the engine silently discarded some sites, the system still believed the retrieval was complete. Later, source coverage verification of the returned results was added, so that incomplete coverage could be detected and located.

[0072] Several components of this invention can be replaced by equivalent means, specifically as follows: Site-limited expression: can be a site-limiting operator (site:), an allowed domain list, or other equivalent mechanisms to limit retrieval to a specific site; Single query operator limit: a configurable parameter set according to the search engine's capabilities, which can be declared separately in the system configuration file; Topic matching: can be keyword matching, regular expression matching, or other classifiers, the key point being to tailor relevant sources according to the query content; Concurrency implementation: can employ coroutines, thread pools, or process pools, with each sector's retrieval executor independent of each other; Deduplication key: can be a resource identifier (URL) or its normalized form; Grouping key: can be based on topic, category, or other keys to maintain consistency of sources within the same query block.

[0073] like Figure 6 As shown, this embodiment provides a unique route registration and capacity block fan-out system for multi-source retrieval, including a registry maintenance module, an import period verification module, a source determination module, a deterministic grouping and block module, a regression and merging module, a source coverage verification module, and a wide search volume processing module.

[0074] The registry maintenance module maintains a source registry, registering each mandatory source, its routed voice, applicable level, retrieval availability, and optional category attributes and topic labels for grouping. The import-period verification module, connected to the registry maintenance module, verifies unique routing invariants during the registry import period and prevents the system from entering a query-accepting service state if verification fails. The source determination module, also connected to the registry maintenance module, determines the set of sources to be queried in response to a query, filtering by query mode and applicable level according to the registry. The deterministic grouping and segmentation module, connected to the source determination module, groups the sources retrieved using site-limited operators by category or topic, sorts them according to registry registration order, and divides them into several sector blocks with a predetermined upper limit, each carrying a stable block identifier. The retrieval and merging module, connected to the deterministic grouping and segmentation module, retrieves each sector block as a separate query, merges the results into the same search result set, and removes duplicates by resource locator. The source coverage verification module is connected to the retrieve and merge module, and is used to verify the source coverage of the returned results, detect incomplete coverage, and locate the specific fan-out block and the missing source. The wide search voice part processing module is connected to the source determination module, and is used to directly initiate retrieval for voice parts that are not subject to the upper limit constraint, without segmentation, and merge the results into the same search result set.

[0075] Figure 7 The embodiments of this specification illustrate a hardware structure diagram of an electronic device for a multi-source retrieval unique route registration and capacity block fan-out method. For example... Figure 7 As shown, the electronic device may include at least one processor, memory, RAM, and communication interface, and the at least one processor, memory, RAM, and communication interface are connected together via a bus. At least one processor executes at least one computer-readable instruction stored or encoded in the memory.

[0076] It should be understood that the computer-executable instructions stored in memory, when executed, cause at least one processor to perform the above-described combinations in the various embodiments of this specification. Figures 1 to 6 The description includes various operations and functions.

[0077] In the embodiments of this specification, electronic devices may include, but are not limited to: personal computers, server computers, workstations, desktop computers, laptop computers, notebook computers, mobile electronic devices, smartphones, tablet computers, cellular phones, personal digital assistants, handheld systems, messaging devices, wearable electronic devices, consumer electronic devices, etc.

[0078] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media containing computer-usable program code.

[0079] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described method for unique route registration and capacity block fan-out for multi-source retrieval.

[0080] The present invention also provides a computer program product, including computer program code or instructions, which, when executed on a computer, enable the implementation of a unique route registration and capacity block fan-out method for multi-source retrieval.

[0081] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0082] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A unique route registration and capacity block fan-out method for multi-source retrieval, characterized in that, include: Maintain a source registry, registering each mandatory source and its routed voice, applicable level, retrieval availability status, and optional category attributes and topic labels for grouping; the same mandatory source may have one or more registration records in the registry; Before responding to any query, the method verifies the unique route invariant during the registry loading period: it traverses all mandatory sources in the registry, and if a duplicate route exists, it determines that the import has failed, outputs the source identifier of the duplicated route and its component identifiers, and prevents the method from entering a query-responsive state; the following steps are executed only after the verification passes. In response to a query, the query mode parameters carried in the query request are read, and the system filters according to the query mode and applicable level based on the registry, excluding those whose available status is "not connected," to determine the set of sources to be queried this time. Each source in the set of sources to be queried this time is retrieved only by the voice department registered in the registry, and not by other voice departments. The same source does not appear in the current dispatch plan of two or more voice departments. For a voice segment retrieved using site-qualified operators, the sources routed to that voice segment are grouped according to their category or topic. Those carrying category attributes are grouped according to the value of their category attributes; those carrying topic tags but not category attributes are grouped according to the first value in their topic tags; those carrying neither category nor topic tags are grouped into a single default group; each source is grouped into only one group; after sorting each group according to the registration order of each source in the registry, it is divided into several fan-out blocks by the upper limit of the number of site-qualified operators allowed to be carried in a single query, which is pre-configured for that voice segment. Each source corresponds to one site-qualified operator. Based on the number of sources, each source is assigned to exactly one fan-out block, and the number of sources in each block does not exceed the upper limit. Each block carries a stable block identifier consisting of a group, a block number, and an ordered list of sources. Each fan-out block is retrieved as a query sent to that voice part. The results of each block are merged into the same search result set and deduplicated according to the resource locator. For each sector, the sources appearing in the returned results of that sector are compared with the ordered source list contained in the stable block identifier of that sector. If there are sources listed in the ordered source list but not appearing in the returned results, the sector is marked as incompletely covered and its stable block identifier and the missing source identifier are output. For sources registered to voice parts not subject to the upper limit in the current source set to be searched, no segmentation is performed. Instead, a retrieval request is initiated directly for that voice part, and the results are merged into the same search result set.

2. The unique route registration and capacity block fan-out method for multi-source retrieval according to claim 1, characterized in that, The loading period verification unique routing invariant includes: traversing all mandatory sources in the registry and verifying the voice field to which each source is routed; when any source is repeatedly routed by two or more voices, it is determined as an import failure, and the voice identifier of the source that is repeatedly routed is indicated; the import failure prevents the system from entering a running state that can accept queries.

3. The unique route registration and capacity block fan-out method for multi-source retrieval according to claim 1, characterized in that, After determining the set of sources to be searched this time and before grouping and segmenting, topic gating is also included: the topics involved in the current query are detected according to the content of the current query, and sources with topic labels are retained only if their topics are related to the current query, while sources without topic labels are always retained; after topic gating, sources that are not related to the current query are pruned from the set of sources to be searched this time in order to save retrieval quota.

4. The unique route registration and capacity block fan-out method for multi-source retrieval according to claim 1, characterized in that, Each entry in the registry is constructed by verifying the validity of its field values: the domain name field is verified to ensure that its format conforms to the predetermined specifications; the routed voice, applicable level, and retrieved availability fields are verified to ensure that their values ​​belong to their respective predetermined enumeration sets; and the optional category attributes are verified to ensure that their values ​​belong to the predetermined category enumeration set. If any field fails verification, an error is thrown during construction and the creation of the entry is rejected. Once the source entry is constructed, it is stored in the read-only area of ​​the registry. The registry access interface provided to the outside world during runtime only exposes the read method and does not provide any writing, updating or deleting methods for any field of the entry. Any write request for the field of the entry will be rejected and recorded in the audit log.

5. The unique route registration and capacity block fan-out method for multi-source retrieval according to claim 1, characterized in that, The process of dividing a group into several fan-out blocks based on the upper limit of the number of site-limited operators allowed in a single query, which is pre-configured for the voice part, includes: for each group, after the sources are stably sorted according to their registration order in the registry, the count is accumulated sequentially starting from the first source in the group, and the group is divided into blocks when the upper limit is reached, so that the number of sources in each block does not exceed the upper limit; if the upper limit is less than one, an error is thrown, and no unusable empty block plan is generated; each source corresponds to one site-limited operator, and the count is based on the number of sources.

6. The unique route registration and capacity block fan-out method for multi-source retrieval according to claim 1, characterized in that, The stable block identifier includes: the group identifier to which the fan-out block belongs, the block sequence number within the group starting from zero, and an ordered list of sources contained in the block arranged in the registration order; the stable block identifier ensures that when the same group spans multiple blocks due to the number of sources exceeding the upper limit, each block still retains its identifiable identity, and the cross-block results are correctly merged.

7. The unique route registration and capacity block fan-out method for multi-source retrieval according to claim 1, characterized in that, The process of retrieving each fan-out block as a query sent to that sound part includes: instantiating the corresponding retrieval executor for each fan-out block and initiating the retrieval, with each block retrieval being executed in parallel; merging the results of each block into the same search result set and removing duplicates by resource locator includes: merging the search results of all fan-out blocks into the same search result set and removing duplicate hits by resource locator; marking the retrieval as complete only after all fan-out blocks are successfully retrieved; if any fan-out block retrieval fails, an incomplete status is explicitly returned, without presenting partial results as complete.

8. The unique route registration and capacity block fan-out method for multi-source retrieval according to claim 1, characterized in that, The grouping, the sorting within the group, and the segmentation are only input to the registration order of each source in the registry, the grouping attributes of each source, and the pre-configured upper limit; the registry is stored in an order-preserving structure and traversed according to the registration order, and the sorting within the group is a stable sort; thus, when the registry content, the query mode, and the query are the same, the generated fan-out block list and its stable block identifier are identical one by one.

9. The unique route registration and capacity block fan-out method for multi-source retrieval according to claim 1, characterized in that, The voice parts include voice parts retrieved using the site-limited operator method, voice parts retrieved using the allowed domain list method, and voice parts retrieved using the broad search method; for voice parts that are not subject to the upper limit constraint of the site-limited operator, their corresponding sources are not divided into blocks, and the retrieval executor is directly instantiated to initiate retrieval.

10. A unique route registration method for multi-source retrieval, characterized in that, include: Maintain a source registry, registering each mandatory source, its routed voice, applicable level, and retrieval availability status. During the registry loading period, the unique route is not verified: if a duplicate route exists, the import is deemed to have failed, and the method is not allowed to enter a query-responsive state. After the verification is successful, proceed to the following steps; In response to a query, the query mode parameters are read, and the system filters according to the query mode and applicable level based on the registry, excluding those whose available status is "not connected," to determine the set of sources to be queried this time. Each source in the set of sources to be queried this time is retrieved only by the voice department registered in the registry, and not by other voice departments. The same source does not appear in the current dispatch plan of two or more voice departments. The results of retrievals initiated separately by each audio segment are merged into the same search result set and deduplicated according to the resource locator.

11. The unique route registration method for multi-source retrieval according to claim 10, characterized in that, Each source entry in the registry is constructed by verifying the validity of its field values: the domain name field is verified to ensure that its format conforms to the predetermined specifications, and the fields of the routed voice, applicable level, and retrieved availability status are verified to ensure that their values ​​belong to their respective predetermined enumeration sets. If any field fails validation, an error will be thrown during construction and the creation of that entry will be rejected. Once the source entry is constructed, it is stored in the read-only area of ​​the registry. The registry access interface provided to the outside world during runtime only exposes the read method and does not provide any writing, updating or deleting methods for any field of the entry. Any write request for the field of the entry will be rejected and recorded in the audit log.

12. A unique route registration and capacity block fan-out system for multi-source retrieval, characterized in that, include: The registry maintenance module is used to maintain a source registry, registering each mandatory source and its routed voice, applicable level, retrieval availability status, and optional category attributes and topic labels for grouping. The import period verification module is connected to the registry maintenance module and is used to verify the unique routing invariant during the registry import period, and to prevent the system from entering the service state that can receive queries when the verification fails. The source determination module is connected to the registry maintenance module. In response to a query, it filters the data according to the registry based on the query mode and applicable level, and excludes those whose available status is not connected, to determine the set of sources to be queried this time. The deterministic grouping and segmentation module, connected to the source determination module, is used to classify the sources in the current source set that are routed to the source according to their category or theme into one or more groups for the voice part retrieved by the site-limited operator. Each source is only classified into one group, and those without grouping attributes are classified into a single default group. After sorting each group according to the registration order of each source in the registry, it is divided into several fan-out blocks by the upper limit of the number of site-limited operators allowed to be carried in a single query, which is pre-configured for the voice part. Each source corresponds to one site-limited operator. Based on the number of sources, each source that should be queried and routed to the voice part is assigned to exactly one fan-out block. The number of sources contained in each block does not exceed the upper limit. Each block carries a stable block identifier consisting of a group, a block number, and an ordered list of sources. The regression module, connected to the deterministic grouping and segmentation module, is used to retrieve each fan-out block as a query sent to the sound part, merge the results of each block into the same retrieval result set and remove duplicates according to the resource locator. The source coverage verification module, connected to the regression and union module, is used to compare the sources appearing in the returned results of each sector with the ordered source list contained in the stable block identifier of the sector. If there are sources listed in the ordered source list but not appearing in the returned results, the sector is marked as incompletely covered and its stable block identifier and the missing source identifier are output. The wide-search sound part processing module, connected to the source determination module, is used to, for sound parts not subject to the upper limit constraint, directly instantiate the retrieval executor to initiate retrieval without performing the segmentation, and merge the results into the same retrieval result set.

13. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the unique route registration and capacity block fan-out method for multi-source retrieval as described in any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the unique route registration and capacity block fan-out method for multi-source retrieval as described in any one of claims 1 to 11.

15. A computer program product, characterized in that, It includes computer program code or instructions that, when executed on a computer, cause the unique route registration and capacity block fan-out method for multi-source retrieval as described in any one of claims 1 to 11 to be implemented.

Images