Enterprise content retrieval management system based on feature matching

Abstract

The application relates to the technical field of enterprise information retrieval and data processing, in particular to an enterprise-level content retrieval management system based on feature matching, which comprises the following steps: receiving a retrieval request containing original attribute data of a to-be-retrieved object; reading main data and to-be-retrieved auxiliary data from a bottom layer storage area, extracting a preset number of attribute fields arranged according to priorities, generating a retrieval code through binary coding according to field mean values; dividing a three-level cache into a main retrieval area and an auxiliary retrieval area through a bottom layer instruction, fixing the main data retrieval code and sequentially loading the to-be-retrieved auxiliary data retrieval code; completing a preliminary selection by adopting bit-by-bit XOR to count the total number of difference bits; intercepting the preliminary selection result according to the current system concurrency, loading corresponding original attribute data to an independent memory buffer area for similarity calculation, and outputting a retrieval result; the system generates a retrieval code to control a preset reading path, reduces the transmission load of a data bus and the passive elimination rate of the three-level cache.

Description

Technical Field

[0001] This invention relates to the field of enterprise information retrieval and data processing technology, specifically to an enterprise-level content retrieval management system based on feature matching. Background Technology

[0002] As a crucial foundation for information management and business operations, enterprise-level content retrieval systems face increasing complexity in data organization and retrieval request structures as the scale of various content types, including text, images, and videos, continues to expand within enterprises. To ensure retrieval response efficiency and result availability, and to effectively manage frequently accessed data and dynamic data awaiting retrieval, it is typically necessary to balance initial screening speed, cache stability, and subsequent accurate sorting during the retrieval process to meet the content retrieval needs of high-concurrency scenarios.

[0003] However, when performing searches on large-scale enterprise content, the common approach is to directly load the complete original attribute data of candidate data into the processing chain and then perform similarity comparisons. While this method can complete candidate screening and result ranking, the introduction of full original attribute data in the initial screening stage increases the data transfer load between the underlying storage area and the processor. Furthermore, it causes high-frequency data to compete with the data to be searched for cache resources, resulting in cache thrashing and repeated replacement of high-frequency data. Simultaneously, conventional solutions often use a fixed proportion or a fixed number of candidates, which can be expressed as: or This calculation method lacks the ability to dynamically respond to the real-time load status of the system. When the system concurrency exceeds the high load threshold, it triggers the subsequent read load to exceed the processing limit. When the system concurrency is lower than the low load threshold, it reduces the amount of recalled data, affecting the system's preset concurrency throughput indicators. Summary of the Invention

[0004] The purpose of this invention is to provide an enterprise-level content retrieval management system based on feature matching, addressing the following technical problems: existing systems trigger cache state changes and high-frequency data eviction when loading the full amount of raw data, and the fixed configuration of the candidate set size lacks a mechanism for dynamic adjustment based on system load. This invention provides an enterprise-level content retrieval management system based on feature matching that achieves primary and secondary isolation through retrieval code comparison and three-level cache partitioning, and dynamically and adaptively adjusts the candidate set size according to real-time system pressure. The objective of this invention can be achieved through the following technical solutions:

[0005] An enterprise-level content retrieval management system based on feature matching runs on a computing device that includes a central processing unit, a three-level cache, an independent memory buffer, and an underlying storage area, including:

[0006] The request receiving module is used to receive retrieval requests containing the original attribute data of the object to be searched.

[0007] The data processing module is used to read raw attribute data from the underlying storage area, the raw attribute data including main data and auxiliary data to be queried; extract the first preset number of attribute fields in the raw attribute data according to priority, count the value of each attribute field to calculate the field mean, and perform binary encoding processing based on the field mean to generate the corresponding retrieval code;

[0008] The cache control module is used to divide the three-level cache into a main retrieval area and a secondary retrieval area through low-level instructions, fix the retrieval code of the main data in the main retrieval area, and load the retrieval code of the secondary data to be searched into the secondary retrieval area in sequence.

[0009] The initial selection module is used to perform a bitwise XOR comparison between the main data retrieval code in the main retrieval area and the retrieval code to be searched in the auxiliary retrieval area, count the total number of different bits, and determine the initial selection result based on the total number of different bits.

[0010] The reordering module is used to obtain the current system concurrency, calculate the corresponding number of candidates, and extract the preliminary selection result based on the number of candidates; trigger a read instruction to load the original attribute data corresponding to the extracted preliminary selection result from the underlying storage area to the independent memory buffer, perform similarity calculation with the original attribute data of the object to be searched, and output the search result based on the calculation result.

[0011] Optionally, the data processing module includes a field extraction submodule and an encoding submodule;

[0012] The field extraction submodule is used to: extract the top preset number of attribute fields from the original attribute data of the master data and the auxiliary data to be searched, which have a historical retrieval frequency greater than a preset high-frequency threshold, in order of priority.

[0013] The encoding submodule is used to: calculate the average value of all data in each attribute field in the underlying storage area; compare the value of the original attribute data in each attribute field with the corresponding average value; if the value is greater than or equal to the average value, it is recorded as the first value; if the value is less than the average value, it is recorded as the second value; and generate the retrieval code by combining the first value and the second value.

[0014] Optionally, the cache control module includes:

[0015] The resident area construction submodule is used to divide the main retrieval area in the three-level cache, lock the retrieval code of the main data in the main retrieval area, and shield the least recently used eviction policy.

[0016] The sequential area construction submodule is used to allocate an independent space in the three-level cache as the auxiliary retrieval area. When a cross-database type retrieval request is received, it intercepts the loading action of the original attribute data of the auxiliary data to be queried in the underlying storage area and only allows the retrieval code of the auxiliary data to be queried to enter the auxiliary retrieval area.

[0017] Optionally, the initial selection module is used to perform the following steps:

[0018] The vector instruction set of the central processing unit is invoked to read the main data retrieval code in the main retrieval area and the auxiliary data retrieval code to be searched in the auxiliary retrieval area into the register of the central processing unit;

[0019] Perform a bitwise XOR comparison on the two sets of search codes;

[0020] The total number of differences in the statistical comparison results;

[0021] The configuration is as follows: if the total number of difference bits is less than the preset difference bit determination threshold, the auxiliary data retrieval code corresponding to the total number of difference bits will be generated and filtered as the initial selection result; if the total number of difference bits is greater than or equal to the preset difference bit determination threshold, the auxiliary data retrieval code will be removed.

[0022] Optionally, the preliminary selection results are truncated based on the number of candidates, including:

[0023] Obtain the total number of segments in the auxiliary data to be queried;

[0024] Candidate values ​​are calculated based on the preset extraction coefficients used to characterize the basic candidate retention ratio, the total number of segments, the current system concurrency, and the pressure adjustment parameters used to characterize the system load contraction magnitude.

[0025] Compare the set minimum number of candidates with the candidate value, and take the larger of the two as the number of candidates;

[0026] Based on the total number of differences in ascending order, extract the number of preliminary selection results for each candidate.

[0027] Optionally, candidate values ​​are calculated based on a preset base extraction coefficient used to characterize the base candidate retention ratio, the total number of segments, the current system concurrency, and a pressure adjustment parameter used to characterize the system load contraction magnitude. This includes: obtaining the pressure adjustment parameter and the current system concurrency; and determining the attenuation ratio of the total number of segments under the current system concurrency based on the pressure adjustment parameter and the base extraction coefficient, so as to obtain the candidate values.

[0028] Optionally, obtain the current system concurrency, including:

[0029] The rearrangement module obtains the real-time load rate of the computing device;

[0030] Obtain the preset maximum load rate, and perform normalization processing based on the ratio between the real-time load rate and the maximum load rate to determine the current system concurrency.

[0031] Optionally, the rearrangement module includes an independent memory computing submodule;

[0032] The independent in-memory computing submodule is used to perform the following steps:

[0033] Bypassing the main search area, the original attribute data corresponding to the extracted preliminary selection results is directly read from the underlying storage area into the independent memory buffer;

[0034] Based on the spatial vector model, multidimensional features are extracted from the original attribute data of the object to be searched and the original attribute data corresponding to the preliminary selection results, and the spatial angle similarity value between the two is calculated; the angle similarity value is used as the result of similarity calculation.

[0035] Optionally, the search results can be output based on the calculation results, including:

[0036] Obtain the similarity values ​​of the included angles corresponding to all the preliminary selection results after truncation;

[0037] The preliminary selection results are sorted in descending order according to the similarity value of the included angle;

[0038] Extract the top-ranked target number of the initial selection results as target data, and output them as the search results.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1. This system generates retrieval codes by extracting attribute fields from the original attribute data and divides the three-level cache into a main retrieval area and a secondary retrieval area. During the initial selection, the retrieval codes are compared bit by bit using XOR to avoid increasing the data bus transmission load caused by loading the full amount of original data in the initial screening stage. By fixing the main data retrieval codes and loading the secondary data retrieval codes sequentially, the problems of cache jitter and high-frequency data replacement are solved.

[0041] 2. This system dynamically calculates the number of candidates by acquiring the current system concurrency and combining the basic extraction coefficient, total number of segments, and pressure adjustment parameters. This mechanism enables the system to adaptively extract the initial selection results based on the real-time load rate, overcoming the deficiency of lacking dynamic response capability when the fixed extraction scale is used. When the system concurrency is higher than the threshold, the reading load is controlled to be within a preset range, and when the concurrency is lower than the threshold, the target recall data ratio is allocated.

[0042] 3. During the re-sorting process, this system bypasses the main retrieval area and directly reads the original attribute data of the initial selection results into an independent memory buffer; it calculates the similarity value of the angle between the sum of the products of the corresponding fields and the vector magnitude, and outputs the retrieval results in descending order; this method prevents the complete data of the precise sorting from flowing back and crowding the main cache, while performing information compensation during the precise sorting stage, and outputs the target number of sorting results while maintaining the preset isolation distribution of the main and auxiliary caches. Attached Figure Description

[0043] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0044] Figure 1 This is a schematic diagram of a module of an enterprise-level content retrieval management system based on feature matching, provided in an embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0046] like Figure 1 As shown, the feature-matching-based enterprise-level content retrieval management system runs on a computing device that includes a central processing unit, a three-level cache, an independent memory buffer, and an underlying storage area, including:

[0047] The request receiving module is used to receive retrieval requests containing the original attribute data of the object to be searched.

[0048] The data processing module is used to read raw attribute data from the underlying storage area. The raw attribute data includes master data and auxiliary data to be queried. It extracts the first preset number of attribute fields in the raw attribute data according to priority, counts the values ​​of each attribute field to calculate the field mean, and performs binary encoding processing based on the field mean to generate the corresponding retrieval code.

[0049] The cache control module is used to divide the three-level cache into a main retrieval area and a secondary retrieval area through underlying instructions, fix the retrieval code of the main data in the main retrieval area, and load the retrieval code of the secondary data to be retrieved into the secondary retrieval area in sequence.

[0050] The initial selection module is used to perform a bitwise XOR comparison between the master data retrieval code in the main retrieval area and the retrieval code of the auxiliary data to be searched in the auxiliary retrieval area, count the total number of differences, and determine the initial selection result based on the total number of differences.

[0051] The reordering module is used to obtain the current system concurrency, calculate the corresponding number of candidates, and extract the preliminary selection results based on the number of candidates. It triggers a read command to load the original attribute data corresponding to the extracted preliminary selection results from the underlying storage area to an independent memory buffer, calculates the similarity with the original attribute data of the object to be searched, and outputs the search results based on the calculation results.

[0052] The data processing module includes a field extraction submodule and an encoding submodule;

[0053] The field extraction submodule is used to extract the top preset number of attribute fields from the original attribute data of master data and auxiliary data to be searched, arranged by priority, for master data and auxiliary data whose historical retrieval frequency is greater than a preset high-frequency threshold.

[0054] The encoding submodule is used to: calculate the average value of all data in each attribute field in the underlying storage area; compare the value of the original attribute data in each attribute field with the corresponding average value; if the value is greater than or equal to the average value, it is recorded as the first value; if the value is less than the average value, it is recorded as the second value; and generate a retrieval code by combining the first value and the second value.

[0055] The cache control module includes:

[0056] The resident area construction submodule is used to divide the main retrieval area in the three-level cache, lock the retrieval code of the main data in the main retrieval area, and shield the least recently used eviction policy.

[0057] The sequential area construction submodule is used to allocate an independent space in the three-level cache as an auxiliary retrieval area. When a cross-database type retrieval request is received, it intercepts the loading action of the original attribute data of the auxiliary data to be queried in the underlying storage area and only allows the retrieval code of the auxiliary data to be queried to enter the auxiliary retrieval area.

[0058] The initial selection module is used to perform the following steps:

[0059] The vector instruction set of the central processing unit is invoked to read the main data retrieval code in the main retrieval area and the secondary data retrieval code to be searched in the secondary retrieval area into the register of the central processing unit.

[0060] Perform a bitwise XOR comparison on the two sets of search codes;

[0061] The total number of differences in the statistical comparison results;

[0062] The configuration is as follows: if the total number of differing bits is less than the preset difference bit judgment threshold, the search code of the auxiliary data to be searched corresponding to the total number of differing bits will be selected as the initial result; if the total number of differing bits is greater than or equal to the preset difference bit judgment threshold, the search code of the auxiliary data to be searched will be removed.

[0063] The workflow of the enterprise-level content retrieval management system based on feature matching includes: The system is designed for enterprise content retrieval scenarios. The input is a retrieval request containing the original attribute data of the object to be searched. In the process, a retrieval code is first generated from the main data and the auxiliary data to be searched. Then, a preliminary screening is completed by using a three-level cache partition and bitwise XOR comparison. The initial selection results are output for subsequent review.

[0064] In the initial screening stage, the complete original attribute data of the auxiliary data to be searched is not loaded. Instead, a search code with a specified bit width is used for comparison, thereby reducing the data transfer load between the underlying storage area and the central processing unit and reducing the replacement rate of high-frequency main data in the L3 cache. The search request is obtained, and the original attribute data of the object to be searched carried in the request is used to access and parse the object to be searched to determine the main data set, the auxiliary data set to be searched, and the corresponding search task.

[0065] The specific mechanism for access parsing the target object and determining the main data set and the auxiliary data set to be queried is as follows: extract the data entity tags or business category target identification carried in the retrieval request, compare them with the entity retrieval frequency statistics table pre-maintained by the system, classify the data entities whose corresponding historical retrieval frequency is higher than the specified frequency threshold into the main data set, and classify the remaining low-frequency data entities into the auxiliary data set to be queried.

[0066] The retrieval request, as a data object written into the address space of this system process by the upper-layer business process, can be read by the request receiving module through object references or structured records in shared memory;

[0067] The original attribute data constitutes the basic field values ​​used for subsequent encoding and similarity judgment; the main data that has been parsed and defined belongs to the data set with high retrieval frequency and needs to be permanently involved in the comparison, while the auxiliary data to be searched refers to the data set that enters the comparison process in sequence during the retrieval process;

[0068] This system defines task boundaries during the request reception phase, ensuring that primary data and secondary data to be queried enter different cache areas in subsequent phases, thus avoiding cache impact caused by indiscriminate full loading. This embodiment can split the retrieval task based on request object parsing, blocking the writing of all original data to the same processing path, and avoiding an increase in the frequency of passive cache eviction due to the lack of primary-secondary isolation mechanism. The system obtains the original attribute data in the underlying storage area, performs field extraction and binary encoding processing on the original attribute data according to preset priority rules, and generates the corresponding retrieval code.

[0069] The specific method for setting the preset priority rules is as follows: Statistically analyze the frequency of each attribute field as a search condition in historical search requests, or for discrete attribute fields, statistically analyze the probability of each value appearing in the full sample data of the underlying storage area, and then apply the formula... Calculate information entropy, where Discrete attribute field The Each possible value The total number of possible values. For the first The probability of each value occurring; for continuous attribute fields, first divide them into several intervals at equal distances and then calculate the information entropy according to the above formula. Attribute fields with frequency of occurrence or information entropy higher than the set threshold are prioritized and arranged in order of index value from high to low.

[0070] The underlying storage area provides the data source as the storage area for storing complete original attribute data. The extracted attribute fields, arranged in order of priority, constitute the set of fields that carry the main distinguishing ability in terms of retrieval semantics. The field mean is the statistical mean of a certain attribute field across all samples in the underlying storage area, which is used to map continuous numerical values ​​into binary bit values.

[0071] In the specific encoding process, the encoding submodule uses the field mean as a unified judgment threshold to obtain the original value of the data sample in the corresponding attribute field. If the original value is greater than or equal to the field mean of all data, the encoded bit value of the field is recorded as the first value. If the original value is less than the field mean, it is recorded as the second value. Then, the bit values ​​of each field are combined in the field order to form the retrieval code. In the specific structured implementation of business logic, in order to avoid synchronously calculating the mean of all data in the real-time retrieval request chain and causing request blocking, the encoding submodule is configured to have an independent background calculation task perform the step of calculating the field mean of all data in each attribute field during system initialization or non-peak business periods.

[0072] For example, the background task traverses the samples in the underlying storage area at fixed intervals to calculate the mean and updates it to the high-speed shared memory. When a retrieval request is received in real time, the latest mean is read directly from the shared memory as a unified judgment threshold to compare the original values. This calculation rule, which decouples the mean calculation from the real-time query, reduces the impact of large-scale full data statistics on the retrieval response time.

[0073] Furthermore, in the specific implementation, the field extraction submodule first determines whether the search frequency exceeds a preset high-frequency threshold. This first preset threshold is used to distinguish between high-frequency data and non-high-frequency data. Its calibration basis is as follows: the system counts the number of times each data record is searched within a historical statistical period, sorts the number of times all records are searched in descending order, and takes the average number of times searched in the first preset position interval, such as the 10% to 20% percentile, as the first preset threshold, thereby ensuring that the extracted master data is kept in the cache. For master data and auxiliary data to be searched that exceed the first preset threshold, only the first preset number of attribute fields are extracted according to priority. If the preset number is not an integer multiple of the bit width of the underlying register, zeros are padded in the low bits to align with the hardware vector length, so as to convert the original high-dimensional features into a structured bit string suitable for single instruction stream execution.

[0074] The system retains key distinguishing information by combining priority field extraction with mean encoding, and transforms high-dimensional floating-point features into retrieval codes suitable for register bit operations. This embodiment can generate retrieval codes based on field priority and field mean, avoiding repeated handling of complete original attribute data in the initial screening stage and avoiding exceeding the preset threshold of processing bus load due to loading full data without differences.

[0075] The primary retrieval area is the region in the L3 cache allocated for the long-term residence of primary data retrieval codes. The secondary retrieval area is the region in the L3 cache allocated for the sequential flow of secondary data retrieval codes. The partitioning operation is implemented by calling processor-level low-level instructions that support cache allocation technology, binding specific L3 cache paths to the primary data process, thereby achieving physical cache isolation. After the primary retrieval area construction module writes the primary data retrieval codes into the primary retrieval area, it keeps them locked and disables the least recently used eviction policy, so that the primary data retrieval codes are not passively removed due to the inflow of secondary data. When the sequential area construction module receives a cross-database type retrieval request, it controls the read path of the underlying storage area, allowing only the retrieval codes of the secondary data to enter the secondary retrieval area, and not allowing the complete original attribute data of the secondary data to occupy the L3 cache first.

[0076] A cross-database retrieval request refers to a retrieval task that requires joint feature comparison of heterogeneous data distributed across different underlying physical storage nodes or data entities involving different data modalities. For joint feature comparison involving different data modalities, a cross-modal mapping table is pre-constructed. The system uses a conventional cross-modal contrastive learning model in this field to construct this cross-modal mapping table. By sharing a semantic space, the original attribute data of heterogeneous data is mapped to a unified feature space of the same dimension, ensuring that the attribute fields of the cross-modal auxiliary data to be queried and the main data have the same feature dimension and physical dimension in the same position. Then, a preset number of attribute fields are extracted to generate retrieval codes.

[0077] The system separates the stable residence of high-frequency primary data from the sequential inflow of secondary data to be queried by using hard partitioning of the cache and primary-secondary isolation, and transforms the high-frequency data access path into a fixed path.

[0078] This embodiment can achieve primary and secondary isolation based on three-level cache partitioning and resident control of retrieval codes, avoiding the encroachment of the complete payload of the secondary data to be queried on the primary data cache space, and finally solving the technical defect of the prior art that high-frequency data is repeatedly eliminated due to the lack of cache partitioning mechanism; obtain the primary data retrieval code in the primary retrieval area and the secondary data retrieval code to be queried in the secondary retrieval area, and perform bitwise XOR comparison of the two sets of retrieval codes according to the vector instruction set of the central processing unit to generate the preliminary selection result;

[0079] During the comparison process, bitwise XOR comparison compares two bit values ​​in the same bit sequence. If they are different, a difference bit is generated; if they are the same, no difference bit is generated. The cumulative number of difference bits in all bit sequences is counted as the total number of difference bits, and this is compared with the judgment boundary used to decide whether to retain or remove candidate records, namely the difference bit judgment threshold. The specific basis for determining the second preset threshold is: the system pre-counts the distribution of difference bits in historical real search samples during bitwise XOR comparison to ensure the target recall rate, such as 95%, as a constraint. The upper limit of the distribution of the number of difference bits corresponding to the target recall rate is rounded down and used as the second preset threshold.

[0080] For example, when the total number of bits in the search code is 64, if historical sample analysis shows that the total number of difference bits is less than 18, it can cover more than 95% of the relevant candidates, then the second preset threshold is configured to 18. The calculation process of the total number of difference bits is as follows: traverse the total number of bits in the search code, perform a bitwise XOR operation on the bit values ​​of the main data search code and the auxiliary data search code to be searched in the same position, and sum the number of times the operation result is 1 to obtain the total number of difference bits. After the initial selection module calls the central processing unit vector instruction set, it reads the search codes in the main search area and the auxiliary search area into the register, and performs XOR and difference bit statistics in the register.

[0081] The specific execution logic is as follows: First, a single instruction multiple data stream instruction performs a bitwise XOR operation on the search codes in the two sets of registers to generate a difference mask. Immediately afterward, a hardware-level population counting instruction is cascaded to directly count the total number of binary bits that are 1 in the mask within one clock cycle, which is used as the total number of difference bits. If the total number of difference bits is less than the second preset threshold, it is selected as the initial result. If the total number of difference bits is greater than or equal to the second preset threshold, it is rejected. The system concentrates the initial screening action between the registers and the L3 cache, so that the auxiliary data to be searched only enters the auxiliary search area in the form of search codes during the initial screening stage, and the full amount of original attribute data is delayed to the subsequent processing steps.

[0082] This embodiment can determine the initial selection result based on bitwise XOR and difference bit threshold judgment, avoiding the premature reading of a large amount of original attribute data in the preliminary screening stage, and ultimately solving the technical defect of the existing technology that causes the bus and cache to be under pressure simultaneously due to reading the full data first and then comparing.

[0083] This implementation is deployed in an enterprise content retrieval server; the business side writes the retrieval request as a structured object into the process memory corresponding to the request receiving module during the system cycle, and the request receiving module generates a retrieval task identifier after reading the object; the data processing module maps and reads the original attribute data of the main data and the auxiliary data to be queried from the underlying storage area, and generates a retrieval code;

[0084] The cache control module completes the three-level cache partitioning through low-level instructions, locks the main data retrieval code into the main retrieval area, and sequentially writes the secondary data retrieval codes to be searched into the secondary retrieval area; the initial selection module calls the central processing unit register to complete the XOR comparison, and writes the record numbers that meet the threshold conditions into the local memory array, so that the subsequent reordering stage can trigger the reading of the underlying storage area according to the number; in this way, the system outputs not only the intermediate comparison results, but also forms a set of candidate numbers that can continue to drive the loading and precise sorting of subsequent original attribute data, thereby completing the actual implementation process from request access to initial selection output;

[0085] In this embodiment, the preliminary selection results are truncated based on the number of candidates, including:

[0086] Get the total number of segments in the auxiliary data to be queried;

[0087] Candidate values ​​are calculated based on the preset extraction coefficients used to characterize the basic candidate retention ratio, the total number of segments, the current system concurrency, and the pressure adjustment parameters used to characterize the system load contraction magnitude.

[0088] Compare the set minimum number of candidates with the candidate value, and take the larger of the two as the number of candidates;

[0089] Based on the total number of differences in ascending order, extract the initial selection results for a certain number of candidates.

[0090] Based on the preset extraction coefficients used to characterize the basic candidate retention ratio, the total number of segments, the current system concurrency, and the pressure adjustment parameters used to characterize the system load contraction magnitude, candidate values ​​are calculated, including: obtaining the pressure adjustment parameters and the current system concurrency; and determining the attenuation ratio of the total number of segments under the current system concurrency based on the pressure adjustment parameters and the basic extraction coefficients, so as to obtain candidate values.

[0091] Get the current system concurrency, including:

[0092] The reordering module obtains the real-time load rate of the computing device;

[0093] Obtain the preset maximum load rate, normalize it according to the ratio between the real-time load rate and the maximum load rate, and determine the current system concurrency.

[0094] The execution steps for determining the number of candidates in the reordering module are as follows: This process follows the aforementioned preliminary selection results. The input is the set of preliminary selection results where the total number of difference bits has been counted and the current concurrency of the system. During processing, the number of candidates is determined based on the total number of segments of the auxiliary data to be checked, the basic extraction coefficient, the pressure adjustment parameter, and the minimum number of candidates. The output is the data extraction result used for subsequent precise sorting.

[0095] The system dynamically shrinks or expands the candidate size according to the current load, so that the amount of raw attribute data that needs to be read from the underlying storage area remains within a controllable range; it obtains the real-time load rate output by the system monitoring module, normalizes the real-time load rate according to the preset maximum load rate, and determines the current system concurrency.

[0096] In practice, the load monitoring component, or system monitoring module, running on the computing device collects the CPU utilization of the computing device within a set time period. Memory usage percentage The disk I / O latency is then normalized by dividing the disk I / O latency by the total cycle time to obtain the I / O load rate. ; Use formula Perform a weighted summation to calculate the real-time load rate, where the weight parameters are configured as follows: , , , calculate It is provided to the reordering module and processed in conjunction with the pre-set normalization upper limit, i.e., the maximum load rate;

[0097] The reordering module can directly read the real-time load rate value maintained by the system monitoring module in shared memory through local object references, and then read the preset maximum load rate. The real-time load rate is divided by the maximum load rate to obtain the ratio. When the ratio is greater than 1, the current system concurrency is set to 1. When the ratio is less than or equal to 1, the ratio is used as the current system concurrency.

[0098] The system uses the ratio of real-time load rate to maximum load rate to form a normalized concurrency, ensuring that the adjustment of the candidate number is consistent with the actual pressure state of the equipment. This embodiment can determine the current system concurrency based on the system monitoring module and normalization calculation, avoiding insufficient sensitivity to system pressure judgment and avoiding deviation of the candidate selection from the target selection range due to the lack of a load parameter input mechanism. The system obtains the total number of pre-divided segments of the auxiliary data to be queried, and calculates the candidate values ​​based on the basic extraction coefficient, the total number of segments, the current system concurrency, and the pressure adjustment parameters to determine the candidate number.

[0099] During the calculation, the total number of segments represents the total number of records in the auxiliary data to be searched after pre-division under the organization method of the search object; the basic extraction coefficient reflects the basic candidate ratio that needs to be retained under normal load conditions; the pressure adjustment parameter controls the extent of candidate value shrinkage when the system load increases; and the minimum candidate quantity ensures that the basic candidate size can still be retained under high load conditions; candidate values The calculation uses an exponential decay relationship: That is, through the basic extraction coefficient The total number of segments pre-divided into auxiliary data to be queried and in terms of natural constants Based on the bottom, the pressure adjustment parameter Concurrency of the current system The negative of the product is the product of the decaying term of the exponent. After processing with the floor function to obtain the preliminary result, it is then combined with the lowest candidate. Compare the values ​​and take the larger one;

[0100] In specific parameter calibration, the basic extraction coefficient The appropriate throughput range for mapped storage devices is typically between 0.01 and 0.05; pressure regulation parameters. Used to shape the intercept slope when system load increases, the value is generally between 1.5 and 3.0, and is obtained from stress testing calibration before system go-live; minimum candidate value. Set the minimum threshold to meet the system's basic recall and display goals, for example, a fixed 500 items;

[0101] Furthermore, in the specific implementation, the product of the pressure regulation parameter and the current system concurrency can be calculated first, and then the attenuation coefficient can be obtained by taking the natural constant as the base and the negative number of the product as the exponent; the basic extraction coefficient, the total number of segments and the attenuation coefficient can be multiplied and the result can be rounded up to obtain the candidate value; the candidate value can be compared with the lowest candidate quantity and the larger value can be taken as the candidate quantity.

[0102] For example, in quantitative deduction: set the total number of segments. It is 100,000. It is 0.02. It is version 2.0. The current system concurrency is 500 when the system is under low load. When the calculated value is 0.2, the attenuation coefficient is: The value is approximately 0.67. At this point, the candidate value is rounded up to 1341, which is greater than 500. Therefore, 1341 are truncated to achieve the preset recall rate parameter. When the system is in a high-concurrency state... When the value is 1.0, the attenuation coefficient is... The value is approximately 0.135. At this point, the decay calculation result of the candidate value drops to 270, triggering the bottom line protection logic. Finally, the larger value is taken to truncate 500 candidate records, thereby achieving dynamic adaptive truncation under different pressure conditions.

[0103] The system adaptively adjusts the size of the candidate set according to the system pressure by coordinating changes in the basic extraction coefficients, concurrency, and pressure adjustment parameters: when the system concurrency is below a set threshold, the proportion of candidate records truncated is increased; when the system concurrency is above a set threshold, the proportion of candidate records truncated is decreased, so as to reduce the amount of subsequent raw attribute data read.

[0104] This embodiment can dynamically determine the number of candidates based on exponential decay calculation, avoiding the risk of wasting reading resources and compressing the recall space under different working conditions with a fixed threshold, and avoiding the additional computing resource occupation or insufficient effective recall record volume triggered by a fixed truncation ratio; obtain the preliminary selection results and the total number of corresponding difference bits, and truncate the preliminary selection results in sequence according to the number of candidates to generate a candidate record set;

[0105] During the truncation process, records are processed in ascending order of the total number of differing bits, ensuring that records with fewer differing bits and lower inconsistencies between search codes are prioritized. This generates a set of candidate record numbers that have undergone quantity control and are ready to read the complete original attribute data. The rearrangement module reads the initial selection results and their total number of differing bits from the local memory array, then performs sorting or ordered traversal to truncate a number of candidate records from smallest to largest. The numbers of these records are written to another memory array as input containers for subsequent read instructions. The record numbers in this array are released after being consumed by subsequent read operations.

[0106] This implementation model combines the total number of segments and the concurrency level. Under high load, it controls the candidate size with a decay coefficient and releases more candidate space with a basic extraction coefficient under low load. Therefore, it is more suitable for situations where the data size changes significantly in enterprise content retrieval. During application deployment, the system monitoring module writes the real-time load rate into the monitoring field in shared memory. The reordering module reads this field and performs normalization within the system cycle to obtain the current system concurrency level.

[0107] The record numbers and total number of differences output by the initial selection module are stored in a local memory array. After the reordering module calculates the number of candidates according to the above formula, it extracts the corresponding number of record numbers from the array and encapsulates the set of numbers into a structured instruction object for subsequent reading tasks. After the structured instruction object is written to the task entry corresponding to the independent memory buffer, it can directly trigger the reading action of the underlying storage area, providing a clear data loading range for subsequent accurate sorting.

[0108] In this embodiment, the reordering module includes an independent memory computing submodule;

[0109] The standalone in-memory computing submodule is used to perform the following steps:

[0110] Bypassing the main search area, the original attribute data corresponding to the extracted preliminary selection results is directly read from the underlying storage area into an independent memory buffer;

[0111] Based on the spatial vector model, multidimensional features are extracted from the original attribute data of the object to be searched and the original attribute data corresponding to the preliminary selection results. The spatial angle similarity value between the two is calculated, and the angle similarity value is used as the result of similarity calculation.

[0112] The search results are output based on the calculation results, including:

[0113] Obtain the similarity values ​​of the included angles corresponding to all the preliminary selection results after cropping;

[0114] The preliminary selection results are sorted in descending order of the similarity value of the included angle;

[0115] Extract the top-ranked target number of initial selection results as target data and output them as search results.

[0116] The steps for the comparison calculation and result output of the rearrangement module are as follows: This process takes the aforementioned candidate record set as input, which is the number of the preliminary selection result that has been extracted, the original attribute data of the object to be searched, and the original attribute data of the corresponding candidate record in the underlying storage area. During the processing, the candidate original attribute data is read into an independent memory buffer by bypassing the main retrieval area, and then calculated and sorted according to the similarity value of the included angle, and the final retrieval result is output.

[0117] The preliminary screening stage and the precise sorting stage use different data payloads and different storage paths. The former relies on search codes and cache partitions, while the latter only reads the complete original attribute data for a limited number of candidates, thereby reducing the problem of interference in the main search area. The truncated preliminary selection result number is obtained, and the candidate original attribute data in the underlying storage area is read in a targeted manner according to the storage mapping relationship corresponding to the number and loaded into an independent memory buffer.

[0118] This targeted reading operation bypasses the main retrieval area, so that the candidate original attribute data no longer enters the cache path used for the main data retrieval code during the precise sorting stage. Instead, it is directly read as the complete data payload required for reconstruction into an independent memory buffer specifically used to receive the candidate original attribute data and perform similarity calculation. The reordering module writes the truncated preliminary selection result number into the read instruction object. The underlying storage area reads the corresponding original attribute data according to the number and writes the read data block into the independent memory buffer.

[0119] The written data blocks exist in the form of original values, and the corresponding buffer space can be released after the similarity calculation is completed. The system delays the loading of complete original attribute data until the candidates have been shrunk, and then writes it directly into an independent memory buffer, thereby avoiding the backflow of precisely sorted data from interfering with the resident state of the main data retrieval code.

[0120] This embodiment can achieve targeted loading of candidate original attribute data based on an independent memory buffer, avoiding the complete original attribute data from crowding out the main retrieval area again, and ultimately solving the technical defect of the prior art that the cache stability decreases due to the shared path between preliminary screening and precise sorting; it obtains the original attribute data of the object to be searched and the original attribute data of the candidate records, and calculates the angle similarity value according to the sum of the products of the corresponding fields and the vector magnitude to generate the similarity calculation result;

[0121] During the calculation process, the overall length of the original attribute data of the object to be searched, i.e. the first vector magnitude, is multiplied by the overall length of the original attribute data of the candidate record, i.e. the second vector magnitude. The product is used as the denominator to eliminate the difference in vector scale. At the same time, the sum of the products of the original attribute data of the object to be searched and the original attribute data of the candidate record in each corresponding field is calculated.

[0122] similarity of included angles The calculation relationship is as follows: The vector corresponding to the object to be searched In the The original values ​​in each corresponding field The vector corresponding to the candidate record Original values ​​in the same field Multiply each item and sum them to get the sum of the products of the corresponding fields. ,in This represents the total number of dimensions of the corresponding fields used in the comparison between the object to be searched and the candidate records;

[0123] Divide it by the magnitude of the first vector With the second vector magnitude The independent memory calculation submodule first calculates the sum of the products of the corresponding fields, then calculates the first vector magnitude and the second vector magnitude respectively, and uses their product as the denominator; if the denominator is zero, the angle similarity value is set to zero; if the denominator is not zero, the sum of the products of the corresponding fields is divided by the denominator to obtain the angle similarity value, and the angle similarity value is used as the similarity calculation result.

[0124] In the data flow definition of business logic, since the first vector magnitude depends only on the object to be searched and remains constant in a single search task, the system optimizes the calculation rule as follows: when processing the first preliminary result, the first vector magnitude is calculated independently and temporarily stored as a scalar constant. When traversing other preliminary results in subsequent iterations, only the corresponding second vector magnitude and the sum of the products are calculated, and the temporarily stored scalar constant is directly called as the multiplier to generate the denominator. Taking the selection of 100 candidates as an example, the first vector magnitude of the object to be searched only needs to be calculated once instead of 100 times, skipping the repetitive calculation steps in a single search task.

[0125] After candidate narrowing, the complete original attribute data is reintroduced for angle similarity calculation. The complete feature dimension of the original attribute data is introduced by calculating the angle similarity value.

[0126] This embodiment can calculate the angle similarity value based on the complete original attribute data in an independent memory buffer, avoiding semantic distortion caused by directly using the binary preliminary screening results for the final sorting, and reducing the sorting error caused by directly outputting the XOR comparison preliminary selection results; obtain the angle similarity value corresponding to all candidate records, sort the preliminary selection results in descending order according to the size of the angle similarity value, and determine the search results;

[0127] During this process, the preliminary selection results of the top-ranked preset number are extracted as target data. This target data is the set of records that are ranked first after being sorted by the similarity value of the included angle and are used to return to the front end or the upper-level caller. The preset number is determined by dynamically assigning a value based on the single page load parameter issued by the front end or the upper-level caller.

[0128] If no explicit parameters are provided, the preset number is set to the single-page loading threshold configured by the system, such as a fixed integer between 20 and 50, to ensure that the output size of the accurate sorting results precisely matches the display needs of the business side and avoids redundant transmission of invalid data at the network layer or application side; the re-sorting module reads the angle similarity values ​​of all candidate records from an independent memory buffer or local result array, sorts them in descending order of angle similarity values, extracts the top preset number of preliminary results as target data; the target data is encapsulated into a structured result object and output as the search results;

[0129] This implementation first narrows down the candidate range using the preceding steps, then calculates the angle similarity value for the limited candidates, and outputs the results in sorted order based on the angle similarity value. Therefore, it can improve the final sorting quality while keeping the reading scale under control. In specific deployment, after receiving the candidate record number set written in the previous stage, the reordering module constructs the corresponding read instruction object and triggers the underlying storage area to write the candidate's original attribute data into an independent memory buffer. After the independent memory calculation submodule completes the angle similarity value calculation, it writes the number and angle similarity value of each candidate record into the local result array.

[0130] After sorting, the system encapsulates the top-ranked target data into a structured search result object and returns it to the business system by calling the upper-level search interface. The business system then uses this object to display the results or make subsequent calls. Thus, the application process from candidate reconstruction and similarity calculation to result output is fully implemented.

[0131] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. An enterprise-level content retrieval management system based on feature matching, running on a computing device including a central processing unit, a three-level cache, an independent memory buffer, and an underlying storage area, characterized in that: include: The request receiving module is used to receive retrieval requests containing the original attribute data of the object to be searched. The data processing module is used to read raw attribute data from the underlying storage area, the raw attribute data including main data and auxiliary data to be queried; extract the first preset number of attribute fields in the raw attribute data according to priority, count the value of each attribute field to calculate the field mean, and perform binary encoding processing based on the field mean to generate the corresponding retrieval code; The cache control module is used to divide the three-level cache into a main retrieval area and a secondary retrieval area through low-level instructions, fix the retrieval code of the main data in the main retrieval area, and load the retrieval code of the secondary data to be searched into the secondary retrieval area in sequence. The initial selection module is used to perform a bitwise XOR comparison between the main data retrieval code in the main retrieval area and the retrieval code to be searched in the auxiliary retrieval area, count the total number of different bits, and determine the initial selection result based on the total number of different bits. The reordering module is used to obtain the current system concurrency, calculate the corresponding number of candidates, and extract the preliminary selection result based on the number of candidates; trigger a read instruction to load the original attribute data corresponding to the extracted preliminary selection result from the underlying storage area to the independent memory buffer, perform similarity calculation with the original attribute data of the object to be searched, and output the search result based on the calculation result; The preliminary selection results are extracted based on the number of candidates, including: Obtain the total number of segments in the auxiliary data to be queried; Candidate values ​​are calculated based on the preset extraction coefficients used to characterize the basic candidate retention ratio, the total number of segments, the current system concurrency, and the pressure adjustment parameters used to characterize the system load contraction magnitude. Compare the set minimum number of candidates with the candidate value, and take the larger of the two as the number of candidates; Based on the total number of differences in ascending order, extract the number of preliminary selection results for each candidate.

2. The enterprise-level content retrieval management system based on feature matching according to claim 1, characterized in that, The data processing module includes a field extraction submodule and an encoding submodule; The field extraction submodule is used to: extract the top preset number of attribute fields from the original attribute data of the master data and the auxiliary data to be searched, which have a historical retrieval frequency greater than a preset high-frequency threshold, in order of priority. The encoding submodule is used to: calculate the average value of all data in each attribute field in the underlying storage area; compare the value of the original attribute data in each attribute field with the corresponding average value; if the value is greater than or equal to the average value, it is recorded as the first value; if the value is less than the average value, it is recorded as the second value; and generate the retrieval code by combining the first value and the second value.

3. The enterprise-level content retrieval management system based on feature matching according to claim 1, characterized in that, The cache control module includes: The resident area construction submodule is used to divide the main retrieval area in the three-level cache, lock the retrieval code of the main data in the main retrieval area, and shield the least recently used eviction policy. The sequential area construction submodule is used to allocate an independent space in the three-level cache as the auxiliary retrieval area. When a cross-database type retrieval request is received, it intercepts the loading action of the original attribute data of the auxiliary data to be queried in the underlying storage area and only allows the retrieval code of the auxiliary data to be queried to enter the auxiliary retrieval area.

4. The enterprise-level content retrieval management system based on feature matching according to claim 1, characterized in that, The initial selection module is used to perform the following steps: The vector instruction set of the central processing unit is invoked to read the main data retrieval code in the main retrieval area and the auxiliary data retrieval code to be searched in the auxiliary retrieval area into the register of the central processing unit; Perform a bitwise XOR comparison on the two sets of search codes; The total number of differences in the statistical comparison results; The configuration is as follows: if the total number of difference bits is less than the preset difference bit determination threshold, the auxiliary data retrieval code corresponding to the total number of difference bits will be generated and filtered as the initial selection result; if the total number of difference bits is greater than or equal to the preset difference bit determination threshold, the auxiliary data retrieval code will be removed.

5. The enterprise-level content retrieval management system based on feature matching according to claim 1, characterized in that, Based on a preset base extraction coefficient used to characterize the basic candidate retention ratio, the total number of segments, the current system concurrency, and a pressure adjustment parameter used to characterize the system load contraction magnitude, candidate values ​​are calculated, including: obtaining the pressure adjustment parameter and the current system concurrency; and based on the pressure adjustment parameter and the base extraction coefficient, determining the attenuation ratio of the total number of segments under the current system concurrency to obtain the candidate values.

6. The enterprise-level content retrieval management system based on feature matching according to claim 1, characterized in that, Get the current system concurrency, including: The rearrangement module obtains the real-time load rate of the computing device; Obtain the preset maximum load rate, and perform normalization processing based on the ratio between the real-time load rate and the maximum load rate to determine the current system concurrency.

7. The enterprise-level content retrieval management system based on feature matching according to claim 1, characterized in that, The reordering module includes an independent memory computing submodule; The independent in-memory computing submodule is used to perform the following steps: Bypassing the main search area, the original attribute data corresponding to the extracted preliminary selection results is directly read from the underlying storage area into the independent memory buffer; Based on the spatial vector model, multidimensional features are extracted from the original attribute data of the object to be searched and the original attribute data corresponding to the preliminary selection results, and the spatial angle similarity value between the two is calculated; the angle similarity value is used as the result of similarity calculation.

8. The enterprise-level content retrieval management system based on feature matching according to claim 7, characterized in that, The search results are output based on the calculation results, including: Obtain the similarity values ​​of the included angles corresponding to all the preliminary selection results after truncation; The preliminary selection results are sorted in descending order according to the similarity value of the included angle; Extract the top-ranked target number of the initial selection results as target data, and output them as the search results.

Images