A method and system for accelerating audit vector retrieval

Abstract

This invention relates to the field of information retrieval technology and proposes a method and system for accelerating audit vector retrieval. The method includes: vectorizing audit records to obtain an audit vector set; determining dynamic optimization parameters for the audit vector set based on the data distribution density and vector magnitude distribution characteristics; establishing cross-layer neighbor connections for the distance relationships between vectors in the audit vector set, and constructing a hierarchical navigable small-world graph of the audit vector set by combining the temporal correlation characteristics of the audit vector set; receiving query vectors and performing hierarchical parallel matching in the hierarchical navigable small-world graph to obtain candidate audit vector subsets of the audit vector set; weighting and sorting the similarity and confidence of the candidate audit vector subsets, and returning the audit records to the requester according to the sorting results, generating fast audit retrieval records for the original audit dataset; this invention can improve the efficiency of audit vector retrieval.

Description

Technical Field

[0001] This invention relates to the field of information retrieval technology, and in particular to a method and system for accelerating audit vector retrieval. Background Technology

[0002] In the current field of audit data retrieval, audit records encompass both structured fields and unstructured text information. Traditional vectorized encoding methods are not adapted to columnar storage formats for integrated encoding processing, making it difficult to improve the efficiency of audit vector generation and storage / retrieval. The organizational form of vector data cannot meet the high-efficiency read / write requirements of audit retrieval. Furthermore, the current audit vector index construction process does not dynamically adjust parameters based on data distribution density and vector magnitude distribution characteristics. The index structure parameters remain fixed, failing to match the actual distribution characteristics of the audit vector set, resulting in significant deficiencies in the adaptability and operational stability of the index structure.

[0003] Current audit vector retrieval systems generally employ a single-layer index retrieval architecture, failing to construct a hierarchical navigation structure based on the temporal correlation characteristics of audit data. Neighbor connections between vectors are not cross-layered or time-series bound, hindering hierarchical parallel matching operations in the retrieval process. Traditional retrieval results are filtered and sorted using only a single dimension, without incorporating similarity and confidence scores for weighted ranking. This results in insufficient accuracy and low overall retrieval efficiency, failing to support the rapid generation and efficient retrieval of audit data reports. Summary of the Invention

[0004] This invention provides a method and system for accelerating audit vector retrieval to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides a method for accelerating audit vector retrieval, comprising: P1. Based on the columnar storage format, the audit records of the original audit dataset are vectorized and encoded to obtain the audit vector set of the original audit dataset; P2. Based on the data distribution density and vector magnitude distribution characteristics of the audit vector set, determine the dynamic optimization parameters of the audit vector set; P3. Establish cross-layer neighbor connections for the distance relationships between vectors in the audit vector set, and construct a hierarchical navigable small-world graph of the audit vector set by combining the temporal correlation characteristics of the audit vector set. P4. Receive the query vector and perform hierarchical parallel matching in the hierarchical navigable small world graph to obtain a subset of candidate audit vectors for the audit vector set; P5. The similarity and confidence scores of the candidate audit vector subsets are weighted and sorted, and the audit records are returned to the requester according to the sorting results, generating a fast audit retrieval record of the original audit dataset.

[0006] In a preferred embodiment, the step of vectorizing the audit records of the original audit dataset based on a columnar storage format to obtain an audit vector set of the original audit dataset includes: Parse each audit record in the original audit dataset and integrate the structured field content and unstructured text content in the audit record into a field-value set of the original audit dataset; The structured field content in the field-value set is mapped to categories and values, and the mapped discrete categories are converted into integer codes to generate the first feature sub-vector of the field-value set. Word embedding vectorization is performed on the unstructured text content in the field-value set. The embedded text fragments are then converted into fixed-dimensional semantic vectors to generate the second feature sub-vector of the field-value set. Concatenate the first feature vector and the second feature vector in field order; The concatenated total vector is normalized to obtain the audit vector of the audit records; All audit vectors are written to a memory buffer in columnar storage format to form the audit vector set of the original audit dataset.

[0007] In a preferred embodiment, determining the dynamic tuning parameters of the audit vector set based on the data distribution density and vector magnitude distribution characteristics of the audit vector set includes: Traverse the audit vector set, calculate the average distance between each audit vector and several of its nearest neighbors, and generate the data distribution density level of the audit vector set based on the distribution of the average distance values. The magnitudes of each audit vector in the audit vector set are arranged in order, and the minimum, maximum, and median values ​​of the magnitudes are aggregated into a magnitude distribution feature vector of the audit vector set. The data distribution density level is compared with the modulus distribution feature vector by a threshold comparison; Based on the comparison results, determine the dynamic list parameters of the audit index structure; Based on the range of the difference between the maximum and minimum values ​​in the modulus distribution feature vector, the dynamic list parameters of the audit index structure are adjusted so that the dynamic list parameters are positively correlated with the range of the difference, thus obtaining the dynamic tuning parameters of the audit vector set.

[0008] In a preferred embodiment, determining the dynamic list parameters of the audit index structure based on the comparison results includes: The data distribution density level and modulus dispersion identifier contained in the comparison results are obtained. The modulus dispersion identifier is determined by the range of the difference between the maximum and minimum values ​​in the modulus distribution feature vector. When the data distribution density level is high density level and the modulus dispersion indicator is low dispersion indicator; Set the maximum number of neighbors in the index graph in the dynamic list parameters to the first value level, and set the dynamic list parameters in the index graph construction process to the second value level; When the data distribution density level is low density level or the modulus dispersion is high dispersion; Set the maximum number of neighbors parameter to the third value level and the dynamic list parameter to the fourth value level. The third value level is less than the first value level and the fourth value level is less than the second value level. Package the maximum number of neighbors parameter and the dynamic list parameter into a dynamic list parameter of the audit index structure.

[0009] In a preferred embodiment, establishing cross-layer neighbor connections for the distance relationships between vectors in the audit vector set, and constructing a hierarchical navigable small-world graph of the audit vector set by combining the temporal correlation characteristics of the audit vector set, includes: The audit vector in the audit vector set is used as the current vector to be inserted, and a highest-level layer is assigned to the current vector to be inserted. Starting from the top-level entry node, the audit vector is filtered by distance between it and the existing node vectors in the highest layer to obtain candidate neighbor nodes for the audit vector set; Perform association analysis on the first specified number of candidate neighbor nodes to establish bidirectional neighbor connections for the audit vector set; Audit vectors whose timestamp difference in the audit vector set is less than a preset time period threshold are marked as a homogeneous vector group; Perform internal joins on the same source vector groups to obtain the direct join paths of the audit vector set; Using direct connection paths and audit vectors in the audit vector set as key elements, a hierarchical navigable small-world graph of the audit vector set is constructed.

[0010] In a preferred embodiment, the step of constructing a hierarchical navigable small-world graph of the audit vector set, using direct connection paths and audit vectors in the audit vector set as key elements, includes: Each audit vector in the audit vector set is treated as an independent graph node, and the independent graph node includes the audit vector value and the node identifier. Bidirectional neighbor connections and direct connection paths are used as navigation graph association paths, and the identifiers and connection direction attributes of the two ends of the navigation graph association path are recorded. Based on the order in which the navigation graph associated paths are established, the independent graph nodes are associated with the navigation graph associated paths by attributes to obtain the graph structure of the audit vector set; By organizing nodes at different levels in the graph structure according to their respective levels, a cross-level bidirectional navigation structure for the audit vector set is obtained. The entire navigation graph association path between all nodes in the cross-layer bidirectional navigation structure is defined as a hierarchical navigable small-world graph of the audit vector set.

[0011] In a preferred embodiment, the receiving query vector is subjected to hierarchical parallel matching in a hierarchical navigable small-world graph to obtain a candidate subset of audit vectors for the audit vector set, including: Receive the query vector from the external input, use the query vector as the starting point for retrieval, and start the retrieval process from the top-level entry node of the hierarchical navigable small world graph; The process retrieval starts from the top-level entry node of the hierarchical navigable small world graph, using the query vector received from external input as the starting point. In the process retrieval, a threshold similarity test is performed on the distance between the node vectors of the parallel processing steps to be evaluated in the hierarchical navigable small world graph and the query vector to obtain candidate exit nodes of the hierarchical navigable small world graph. The candidate exit nodes are merged and deduplicated to obtain the set of valid exit nodes in the hierarchical navigable small world graph; Using each node in the effective exit node set as the entry node of the next layer, repeat the above parallel processing and filtering operations, and search downwards in sequence until the bottom layer of the hierarchical navigable small world graph is reached. At the underlying level, block-based parallel matching is performed on all nodes to collect audit vectors corresponding to nodes whose similarity distance between query vectors is less than a preset recall threshold, thus obtaining a subset of candidate audit vectors for the audit vector set.

[0012] In a preferred embodiment, the distance similarity is calculated using the following formula: ; In the formula, For distance similarity, For query vector, Let be the audit vector corresponding to any node in the hierarchical navigable small-world graph. To query the magnitude of the vector, For the magnitude of the audit vector, This represents the total number of fields in the audit vector. For the audit vector One field, To preset importance weight coefficients, As an indicator function, the query vector and the audit vector are in the th... When the values ​​of the fields are the same, the value is 0; when they are different, the value is 1. ∏ is the product operator.

[0013] In a preferred embodiment, the step of weighting and sorting the candidate audit vector subset by similarity and confidence, and returning the audit records to the requester according to the sorting result to generate a fast audit retrieval record for the original audit dataset includes: The distance similarity and confidence scores are weighted and fused to obtain the comprehensive ranking score of the audit vector set; Based on the comprehensive ranking score, select a specified number of candidate audit vectors from the top of the candidate audit vector subset; The extracted raw audit records are organized into a search result list according to the order in the queue, and the search result list is returned to the requester. Simultaneously, the query vector used in this retrieval, the size of the candidate audit vector subset, and the summary of the first record of the sorted candidate vector queue are saved to the audit retrieval log to generate a fast audit retrieval record of the original audit dataset.

[0014] To address the above problems, the present invention also provides an audit vector retrieval acceleration system, the system comprising: The audit vector encoding module is used to vectorize the audit records of the original audit dataset based on the columnar storage format to obtain the audit vector set of the original audit dataset. The dynamic parameter tuning module is used to determine the dynamic tuning parameters of the audit vector set based on the data distribution density and vector magnitude distribution characteristics of the audit vector set. The hierarchical graph construction module is used to establish cross-layer neighbor connections for the distance relationships between vectors in the audit vector set, and to construct a hierarchical navigable small-world graph of the audit vector set by combining the temporal correlation characteristics of the audit vector set. The hierarchical parallel matching module is used to receive query vectors and perform hierarchical parallel matching in a hierarchical navigable small world graph to obtain a subset of candidate audit vectors for the audit vector set. The retrieval weighted output module is used to perform weighted sorting of similarity and confidence in the candidate audit vector subset, and return the audit records to the requester according to the sorting results, generating a fast audit retrieval record of the original audit dataset.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention uses a columnar storage format to complete the vectorized encoding of audit records, integrates structured fields and unstructured text information to generate a standardized audit vector set, optimizes the storage and retrieval efficiency of vector data, determines dynamic tuning parameters by combining the data distribution density and modulus distribution characteristics of audit vectors, improves the adaptability and operational stability of the index structure, and strengthens the navigation support capability of audit vector retrieval by constructing a hierarchical navigable small-world graph that integrates temporal correlation characteristics and establishing cross-layer neighbor connections.

[0016] 2. This invention relies on a hierarchical navigable small-world graph to achieve hierarchical parallel matching, quickly filter out candidate audit vector subsets, integrate similarity and confidence to complete weighted sorting, accurately output compliant audit retrieval results, generate efficient audit retrieval records, comprehensively improve the overall efficiency and accuracy of audit vector retrieval, and stably support the needs for rapid retrieval of audit data and efficient generation of data reports. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating an audit vector retrieval acceleration method according to an embodiment of the present invention. Figure 2 This is a functional block diagram of an audit vector retrieval acceleration system provided in an embodiment of the present invention; The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0019] This application provides a method for accelerating audit vector retrieval. The execution entity of this method includes, but is not limited to, at least one of the following: a server, a terminal, or other electronic devices configured to execute the method provided in this application. In other words, the method for accelerating audit vector retrieval can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0020] Reference Figure 1 The diagram shown is a flowchart illustrating an audit vector retrieval acceleration method according to an embodiment of the present invention. In this embodiment, the audit vector retrieval acceleration method includes: P1. Based on the columnar storage format, the audit records of the original audit dataset are vectorized and encoded to obtain the audit vector set of the original audit dataset; In this embodiment of the invention, the step of vectorizing the audit records of the original audit dataset based on a columnar storage format to obtain the audit vector set of the original audit dataset includes: Parse each audit record in the original audit dataset and integrate the structured field content and unstructured text content in the audit record into a field-value set of the original audit dataset; The structured field content in the field-value set is mapped to categories and values, and the mapped discrete categories are converted into integer codes to generate the first feature sub-vector of the field-value set. Word embedding vectorization is performed on the unstructured text content in the field-value set. The embedded text fragments are then converted into fixed-dimensional semantic vectors to generate the second feature sub-vector of the field-value set. Concatenate the first feature vector and the second feature vector in field order; The concatenated total vector is normalized to obtain the audit vector of the audit records; All audit vectors are written to a memory buffer in columnar storage format to form the audit vector set of the original audit dataset.

[0021] Each audit record stored in the original audit dataset is read line by line. The structured field content and unstructured text content contained in each record are accurately identified and distinguished by the field type identifiers built into the audit record. Each identified content is bound one-to-one with the corresponding field name preset in the audit record to ensure that there are no omissions or mismatches between the content and the field name. All bound fields and corresponding content are combined to form the field-value set of the original audit dataset.

[0022] The system retrieves a pre-defined, multi-round validated structured field category mapping table based on the audit business scenario. It then precisely matches each structured field in the field-value set with the category items in the table, locating the unique corresponding numerical item in the table. Each mapped discrete category is then converted into a globally unique and non-repeating integer code. Strictly following the fixed arrangement order of the original fields in the audit record, all integer codes are sequentially combined and arranged to generate the first feature sub-vector of the field-value set.

[0023] According to the fixed word segmentation length preset by the audit retrieval requirements, the unstructured text content in the field-value group set is split into segments without overlap or omission, ensuring that the length of the split text segments fully meets the preset standard. Each split text segment is converted into a fixed-dimensional numerical vector with a dimension number that perfectly matches the audit retrieval accuracy requirements through semantic feature extraction. The numerical vectors corresponding to all text segments are sequentially connected and combined according to the original word order of the unstructured text to generate the second feature sub-vector of the field-value group set.

[0024] Strictly follow the established arrangement order of the original fields in the audit record, keep the internal dimension order of the first feature sub-vector and the second feature sub-vector unchanged, and seamlessly connect and combine all dimension values ​​of the first feature sub-vector and all dimension values ​​of the second feature sub-vector in sequence to form a spliced ​​total vector with complete dimensions, unified order and no missing dimensions.

[0025] Extract the maximum and minimum values ​​of all dimensions in the concatenated total vector. Based on the extracted maximum and minimum values, perform linear normalization on the values ​​of each dimension in the total vector to convert the original values ​​of each dimension into normalized integer values ​​within the closed interval of 0 to 1, ensuring that all dimension values ​​stably fall within this interval. After the normalization process is completed, the audit vector corresponding to a single audit record is obtained.

[0026] In accordance with the dimension alignment standards and contiguous storage rules required by the columnar storage format, the audit vectors generated from all audit records are sequentially written into a memory buffer with a preset fixed storage capacity according to the original generation time of the audit records. This ensures that the vector storage location is continuous and the dimension alignment is correct. All audit vectors that have successfully completed the write operation and been stored in the memory buffer together constitute the audit vector set of the original audit dataset.

[0027] P2. Based on the data distribution density and vector magnitude distribution characteristics of the audit vector set, determine the dynamic optimization parameters of the audit vector set; In this embodiment of the invention, determining the dynamic optimization parameters of the audit vector set based on the data distribution density and vector magnitude distribution characteristics of the audit vector set includes: Traverse the audit vector set, calculate the average distance between each audit vector and several of its nearest neighbors, and generate the data distribution density level of the audit vector set based on the distribution of the average distance values. The magnitudes of each audit vector in the audit vector set are arranged in order, and the minimum, maximum, and median values ​​of the magnitudes are aggregated into a magnitude distribution feature vector of the audit vector set. The data distribution density level is compared with the modulus distribution feature vector by a threshold comparison; Based on the comparison results, determine the dynamic list parameters of the audit index structure; Based on the range of the difference between the maximum and minimum values ​​in the modulus distribution feature vector, the dynamic list parameters of the audit index structure are adjusted so that the dynamic list parameters are positively correlated with the range of the difference, thus obtaining the dynamic tuning parameters of the audit vector set.

[0028] The step of determining the dynamic list parameters of the audit index structure based on the comparison results includes: The data distribution density level and modulus dispersion identifier contained in the comparison results are obtained. The modulus dispersion identifier is determined by the range of the difference between the maximum and minimum values ​​in the modulus distribution feature vector. When the data distribution density level is high density level and the modulus dispersion indicator is low dispersion indicator; Set the maximum number of neighbors in the index graph in the dynamic list parameters to the first value level, and set the dynamic list parameters in the index graph construction process to the second value level; When the data distribution density level is low density level or the modulus dispersion is high dispersion; Set the maximum number of neighbors parameter to the third value level and the dynamic list parameter to the fourth value level. The third value level is less than the first value level and the fourth value level is less than the second value level. Package the maximum number of neighbors parameter and the dynamic list parameter into a dynamic list parameter of the audit index structure.

[0029] The process iterates through each audit vector in the audit vector set. Using the currently accessed audit vector as the reference object, it selects eight audit vectors that are adjacent to the current audit vector in the memory buffer and whose number is a fixed preset for the audit business. The spatial interval length between the current audit vector and each of the nearest neighbor vectors is determined one by one by the vector space position measurement method. All the measured spatial interval lengths are accumulated, and the accumulated result is divided by the total number of nearest neighbor vectors to obtain the average distance corresponding to the current audit vector. The average distances calculated for all audit vectors are collected and then divided into five fixed value intervals: 0 to 0.2, 0.2 to 0.4, 0.4 to 0.6, 0.6 to 0.8, and 0.8 to 1. The proportion of the average distance in each value interval to the total number of average distances in that interval is counted. Based on the proportion of the average distance in each value interval, the data distribution density level of the audit vector set is divided into three fixed levels: high density, medium density, and low density.

[0030] Extract the modulus value of each audit vector in the audit vector set after normalization. Arrange all modulus values ​​in ascending order from smallest to largest. When the total number of modulus values ​​is odd, directly extract the modulus value at the middle position of the sequence. When the total number of modulus values ​​is even, extract the modulus value at the beginning of the middle position of the sequence. At the same time, extract the minimum and maximum modulus values ​​from the arranged modulus sequences. Combine these three modulus values ​​in a fixed order of minimum, median and maximum to form the modulus distribution feature vector of the audit vector set.

[0031] The system retrieves the density level judgment threshold and the modulus distribution value threshold that are pre-set according to the audit retrieval performance requirements. It then performs a precise matching and comparison between the data distribution density level and the density level judgment threshold of the audit vector set. Finally, it compares the minimum, median, and maximum values ​​in the modulus distribution feature vector with the modulus distribution value threshold item by item, thus completing the threshold comparison operation between the data distribution density level and the modulus distribution feature vector.

[0032] Based on the density level matching results obtained from threshold comparison and the modulus value comparison results, a set of parameter combinations that are completely compatible with the current comparison results are selected from four pre-configured sets of standard index parameter combinations. The selected set of parameter combinations is then formally determined as the dynamic list parameters of the audit index structure.

[0033] Calculate the numerical difference between the maximum and minimum values ​​in the modulus distribution feature vector, and classify the difference into three preset fixed difference intervals: 0 to 5, 5 to 10, and 10 to 15 or more. According to the fixed parameter adjustment range corresponding to the difference interval, the dynamic list parameters of the audit index structure are adjusted synchronously to ensure that the parameter adjustment range and the difference interval range maintain a positive correspondence. The larger the difference interval range, the larger the parameter adjustment range. After the adjustment is completed, the dynamic optimization parameters of the audit vector set are obtained.

[0034] The data distribution density level corresponding to the audit vector set is accurately extracted from the results of threshold comparison. The numerical difference between the maximum and minimum values ​​in the modulus distribution feature vector is calculated. This difference is assigned to a preset fixed difference range of 0 to 5, 5 to 10, and 10 to 15 or more. The modulus dispersion indicator is determined based on the fixed range in which the difference is located. A difference in the range of 0 to 5 is a low dispersion indicator, and a difference in the range of 10 to 15 or more is a high dispersion indicator. This completes the acquisition of the data distribution density level and modulus dispersion indicator in the comparison results.

[0035] The data distribution density level is checked against the preset high density level judgment criteria one by one. The high density level judgment criteria is that the proportion of the number of items with an average distance in the range of 0 to 0.2 exceeds 60%. The modulus dispersion indicator is checked against the preset low dispersion indicator judgment criteria one by one. When both check results meet the requirements, the maximum number of neighbors parameter of the index graph in the dynamic list parameter is set to the preset first value level 32, and the dynamic list parameter in the index graph construction process is set to the preset second value level 64.

[0036] The data distribution density level is checked against the preset low density level judgment criteria one by one. The low density level judgment criteria is that the proportion of the number of items with an average distance between 0.8 and 1 exceeds 30%. The modulus dispersion indicator is checked against the preset high dispersion indicator judgment criteria one by one. When any one of the check results meets the requirements, the maximum number of neighbors parameter of the index map is set to the preset third value level of 16, and the dynamic list parameter in the index map construction process is set to the preset fourth value level of 32. The fixed value corresponding to the third value level is less than the fixed value corresponding to the first value level, and the fixed value corresponding to the fourth value level is less than the fixed value corresponding to the second value level.

[0037] The maximum number of neighbors parameter of the index graph after the value range is set is integrated and encapsulated with the dynamic list parameters in the index graph construction process. The correspondence and configuration attributes of the parameters remain unchanged. The complete parameter combination after integration and encapsulation is the dynamic list parameter of the audit index structure.

[0038] P3. Establish cross-layer neighbor connections for the distance relationships between vectors in the audit vector set, and construct a hierarchical navigable small-world graph of the audit vector set by combining the temporal correlation characteristics of the audit vector set. In this embodiment of the invention, establishing cross-layer neighbor connections for the distance relationships between vectors in the audit vector set, and constructing a hierarchical navigable small-world graph of the audit vector set by combining the temporal correlation characteristics of the audit vector set, includes: The audit vector in the audit vector set is used as the current vector to be inserted, and a highest-level layer is assigned to the current vector to be inserted. Starting from the top-level entry node, the audit vector is filtered by distance between it and the existing node vectors in the highest layer to obtain candidate neighbor nodes for the audit vector set; Perform association analysis on the first specified number of candidate neighbor nodes to establish bidirectional neighbor connections for the audit vector set; Audit vectors whose timestamp difference in the audit vector set is less than a preset time period threshold are marked as a homogeneous vector group; Perform internal joins on the same source vector groups to obtain the direct join paths of the audit vector set; Using direct connection paths and audit vectors in the audit vector set as key elements, a hierarchical navigable small-world graph of the audit vector set is constructed.

[0039] The construction of a hierarchical, navigable small-world graph of the audit vector set, using direct connection paths and audit vectors in the audit vector set as key elements, includes: Each audit vector in the audit vector set is treated as an independent graph node, and the independent graph node includes the audit vector value and the node identifier. Bidirectional neighbor connections and direct connection paths are used as navigation graph association paths, and the identifiers and connection direction attributes of the two ends of the navigation graph association path are recorded. Based on the order in which the navigation graph associated paths are established, the independent graph nodes are associated with the navigation graph associated paths by attributes to obtain the graph structure of the audit vector set; By organizing nodes at different levels in the graph structure according to their respective levels, a cross-level bidirectional navigation structure for the audit vector set is obtained. The entire navigation graph association path between all nodes in the cross-layer bidirectional navigation structure is defined as a hierarchical navigable small-world graph of the audit vector set.

[0040] Select individual audit vectors from the audit vector set in memory storage order as the current vector to be inserted. Based on the preset hierarchical probability allocation fixed rules, assign a unique and non-overlapping highest level to the current vector to be inserted. The total number of levels does not exceed the preset 5 levels. Complete the binding confirmation between the current vector to be inserted and the corresponding highest level.

[0041] Locate the top-level entry node of the established hierarchical navigation structure, use the current vector to be inserted as the calculation benchmark, and accurately calculate the spatial interval length between the vector and all existing node vectors in the highest layer by comparing the vector spatial positions. Select the nodes whose spatial interval length is in the top 20% of all calculated values. All the selected nodes together constitute the candidate neighbor nodes of the audit vector set.

[0042] According to the fixed sorting rule of spatial interval length from small to large, the candidate neighbor nodes are sorted in ascending order, and the top 8 nodes in the sorting result are selected. A bidirectional mutual reference relationship is established between the current vector to be inserted and these nodes to ensure that the nodes at both ends can recognize the other's associated object, thus completing the bidirectional neighbor connection construction of the audit vector set.

[0043] Extract the timestamp value of each audit vector in the audit vector set, calculate the absolute difference between any two audit vector timestamps, and classify all audit vectors whose timestamp difference is less than the preset 30-minute time period threshold. Add a unique source tag to this category to form a source vector group of the audit vector set.

[0044] For all audit vectors within the same source vector group, establish continuous association binding relationships between adjacent nodes in chronological order of timestamps from earliest to latest. Integrate and summarize all continuous binding relationships within the same source vector group to form a direct connection path for the audit vector set.

[0045] All audit vectors in the audit vector set are taken as independent basic nodes, and the bidirectional neighbor connections and direct connection paths built in the early stage are taken as the association paths between nodes. Based on the hierarchical division standard of the highest level corresponding to each independent basic node, all independent basic nodes and association paths are arranged and integrated in layers to finally build a hierarchical navigable small world graph of the audit vector set.

[0046] Each audit vector in the audit vector set is extracted one by one. A globally unique and non-repeating digital node identifier is assigned to each audit vector. All the dimensional values ​​corresponding to the audit vector are integrated and encapsulated with the unique digital node identifier, so that each encapsulated unit is an independent graph node. The independent graph node fully contains the two core contents of the audit vector value and the node identifier.

[0047] The bidirectional neighbor connections and direct connection paths that were previously established are uniformly classified as navigation graph associated paths. For each navigation graph associated path, the node identifiers of the independent graph nodes connected to both ends of the path are accurately recorded. At the same time, the fixed bidirectional connection direction attribute corresponding to the path is marked, thus completing the complete recording of node identifiers and connection direction attributes for all navigation graph associated paths.

[0048] According to the actual generation order of the navigation map associated paths, each navigation map associated path is fixedly bound and matched with the independent graph nodes at both ends of the path, so that the independent graph nodes and the navigation map associated paths form an unchangeable attribute association relationship. All independent graph nodes that have completed the attribute association are combined with the navigation map associated paths to form the graph structure of the audit vector set.

[0049] Based on the highest level information of the independent graph node, the hierarchical affiliation of each independent graph node in the graph structure is accurately identified. Independent graph nodes of the same level are collected and integrated with the corresponding navigation graph associated paths. Then, according to the fixed rules from high to low level, the content of different levels is arranged in an orderly manner. After collaborative organization, a cross-level bidirectional navigation structure of the audit vector set is obtained.

[0050] The entire independent graph nodes contained within the cross-layer bidirectional navigation structure, as well as all navigation graph association paths between all nodes, are integrated and aggregated, keeping the association relationship between nodes and paths unchanged. The integrated complete structure is then uniformly named and ultimately defined as a hierarchical navigable small-world graph of the audit vector set.

[0051] P4. Receive the query vector and perform hierarchical parallel matching in the hierarchical navigable small world graph to obtain a subset of candidate audit vectors for the audit vector set; In this embodiment of the invention, the receiving of query vectors, performing hierarchical parallel matching in a hierarchical navigable small-world graph to obtain a subset of candidate audit vectors for the audit vector set, includes: Receive the query vector from the external input, use the query vector as the starting point for retrieval, and start the retrieval process from the top-level entry node of the hierarchical navigable small world graph; The process retrieval starts from the top-level entry node of the hierarchical navigable small world graph, using the query vector received from external input as the starting point. In the process retrieval, a threshold similarity test is performed on the distance between the node vectors of the parallel processing steps to be evaluated in the hierarchical navigable small world graph and the query vector to obtain candidate exit nodes of the hierarchical navigable small world graph. The candidate exit nodes are merged and deduplicated to obtain the set of valid exit nodes in the hierarchical navigable small world graph; Using each node in the effective exit node set as the entry node of the next layer, repeat the above parallel processing and filtering operations, and search downwards in sequence until the bottom layer of the hierarchical navigable small world graph is reached. At the underlying level, block-based parallel matching is performed on all nodes to collect audit vectors corresponding to nodes whose similarity distance between query vectors is less than a preset recall threshold, thus obtaining a subset of candidate audit vectors for the audit vector set.

[0052] The formula for calculating the distance similarity is as follows: ; In the formula, For distance similarity, For query vector, Let be the audit vector corresponding to any node in the hierarchical navigable small-world graph. To query the magnitude of the vector, For the magnitude of the audit vector, This represents the total number of fields in the audit vector. For the audit vector One field, To preset importance weight coefficients, As an indicator function, the query vector and the audit vector are in the th... When the values ​​of the fields are the same, the value is 0; when they are different, the value is 1. ∏ is the product operator.

[0053] The system receives the query vector corresponding to the audit retrieval request from the external device. This query vector is generated using the same columnar storage vectorization encoding rule as the audit vector. The query vector is fixed as the starting point of the entire retrieval process. The system accurately locates the preset top-level unique entry node in the hierarchical navigable small world map and officially starts the entire retrieval execution operation from this top-level entry node.

[0054] Using the externally input query vector as the core starting point of the entire retrieval process, and relying on the top-level entry node of the hierarchical navigable small world graph, a complete process retrieval operation covering all nodes and associated paths of the top-level independent graph is carried out along the navigation graph association paths that have been established between the nodes of the top-level independent graph.

[0055] During the continuous execution of the process retrieval, all node vectors to be evaluated in the hierarchical parallel computing stage within the hierarchical navigable small world graph are selected. By combining the judgment methods of vector space association, magnitude difference, and field consistency, the distance similarity between each node vector to be evaluated and the query vector is determined one by one. All nodes with a distance similarity reaching the preset screening standard of 0.75 are retained. All retained nodes together constitute the candidate exit nodes of the hierarchical navigable small world graph.

[0056] All candidate exit nodes generated in the top-level process retrieval stage are aggregated, and the globally unique node identifier of each node is checked one by one. Redundant nodes with completely duplicate identifier information are deleted, and all remaining unique nodes are integrated to form a set of effective exit nodes in a hierarchical navigable small world graph.

[0057] Each independent node in the effective exit node set is used as a dedicated entry node for the next level of the hierarchical navigable small world graph. The node parallel processing and distance similarity filtering operations are repeatedly executed, and the retrieval process is advanced layer by layer in a fixed order from high to low, until the retrieval reaches the bottom level of the hierarchical navigable small world graph.

[0058] At the lowest level of the hierarchical navigable small world graph, each block contains a fixed number of 16 nodes. All independent graph nodes at the bottom level are divided into several independent node blocks. Parallel matching calculations are performed on the nodes within each node block. The similarity distance between the node vector and the query vector is checked one by one. Nodes with a similarity distance less than a preset recall threshold of 0.6 are collected. The audit vectors bound to these nodes are extracted. All successfully extracted audit vectors together constitute a candidate audit vector subset of the audit vector set.

[0059] The query vector is generated from an externally initiated audit retrieval request. It is generated after completing the encoding process according to the columnar storage vectorization encoding process consistent with the original audit records. The encoding process integrates structured fields and unstructured text features simultaneously.

[0060] The audit vectors are extracted precisely according to the node identifiers from the audit vector set formed by columnar storage vectorization encoding and normalization of the original audit dataset.

[0061] The modulus of the query vector is calculated by traversing all the dimension values ​​of the query vector, calculating the square root of the sum of the squares of the dimension values, and then using a length normalization method.

[0062] The magnitude of the audit vector is calculated by traversing all the dimensions of the audit vector, calculating the square root of the sum of the squares of the dimensions, and then using a length normalization method.

[0063] The total number of fields in the audit vector is determined by directly adding the number of structured fields and the number of unstructured text fields after parsing the original audit records.

[0064] The field consistency comparison result is obtained by checking the values ​​of the corresponding fields in the query vector and the audit vector field by field, and determining whether they are the same or different.

[0065] The field importance weight coefficient is determined based on the field priority configuration rules preset in the audit business scenario, assigning higher weights to core audit fields and standard weights to ordinary fields.

[0066] This calculation method comprehensively and accurately measures the matching degree between query vectors and audit vectors by integrating three dimensions: vector space association, vector magnitude difference, and field value consistency. It provides a unified and quantifiable similarity judgment standard for hierarchical parallel matching operations within a layered, navigable small world graph, ensuring the accuracy and consistency of candidate exit node selection. This method is fully adaptable to the field characteristics of audit data, which integrates structured and unstructured data, and aligns with the specific retrieval needs of audit business, effectively improving the accuracy and retrieval rationality of parallel matching at lower-level nodes. Furthermore, this method can continuously and stably output quantifiable matching result values, providing reliable numerical basis for the selection of candidate audit vector subsets and subsequent weighted ranking, supporting the entire process of rapid audit retrieval.

[0067] P5. The similarity and confidence scores of the candidate audit vector subsets are weighted and sorted, and the audit records are returned to the requester according to the sorting results, generating a fast audit retrieval record of the original audit dataset.

[0068] In this embodiment of the invention, the step of weighted sorting of similarity and confidence in the candidate audit vector subset, and returning the audit records to the requester according to the sorting result to generate a fast audit retrieval record for the original audit dataset, includes: The distance similarity and confidence scores are weighted and fused to obtain the comprehensive ranking score of the audit vector set; Based on the comprehensive ranking score, select a specified number of candidate audit vectors from the top of the candidate audit vector subset; The extracted raw audit records are organized into a search result list according to the order in the queue, and the search result list is returned to the requester. Simultaneously, the query vector used in this retrieval, the size of the candidate audit vector subset, and the summary of the first record of the sorted candidate vector queue are saved to the audit retrieval log to generate a fast audit retrieval record of the original audit dataset.

[0069] According to the fixed weighting ratio rules pre-set in the audit retrieval business, the distance similarity value is weighted by a fixed ratio of 70%, and the confidence value is weighted by a fixed ratio of 30%. The weighted distance similarity value and the confidence value are then summed. The final value obtained by the summation is used as the comprehensive ranking score for a single candidate audit vector. The comprehensive ranking scores calculated for all candidate audit vectors are then combined to form the comprehensive ranking score of the audit vector set.

[0070] All audit vectors in the candidate audit vector subset are sorted in descending order according to their corresponding comprehensive ranking scores. The audit vectors that are at the top of the ranking results and whose number is exactly the same as the number of ten returned results preset by the audit retrieval are selected to complete the precise screening of the specified number of candidate audit vectors at the beginning of the candidate audit vector subset.

[0071] Based on the globally unique node identifiers bound to the selected number of candidate audit vectors, the original audit records corresponding one-to-one with the node identifiers are accurately retrieved from the storage location of the original audit dataset. The retrieved original audit records are arranged in order according to the sorting queue of the candidate audit vectors. All the arranged original audit records are combined to form a standardized search result list. The search result list is sent to the requester who initiated this audit search request through a dedicated data transmission channel.

[0072] The complete numerical content of the query vectors used in this retrieval process, the total number of audit vectors actually included in the candidate audit vector subset, and the core field summary information of the first five records of the sorted candidate vector queue are stored sequentially into the pre-divided dedicated storage unit for audit retrieval logs according to a fixed storage format. The log data after all information is stored is the fast audit retrieval record of the original audit dataset.

[0073] like Figure 2 The diagram shown is a functional block diagram of an audit vector retrieval acceleration system provided in an embodiment of the present invention.

[0074] The audit vector retrieval acceleration system 100 described in this invention can be installed in an electronic device. Depending on the functions implemented, the audit vector retrieval acceleration system 100 may include an audit vector encoding module 101, a dynamic parameter tuning module 102, a hierarchical graph construction module 103, a hierarchical parallel matching module 104, and a retrieval weighted output module 105. The module described in this invention can also be called a unit, referring to a series of computer program segments that can be executed by the processor of an electronic device and perform a fixed function, stored in the memory of the electronic device.

[0075] In this embodiment, the functions of each module / unit are as follows: The audit vector encoding module 101 is used to vectorize the audit records of the original audit dataset based on the columnar storage format to obtain the audit vector set of the original audit dataset; The dynamic parameter tuning module 102 is used to determine the dynamic tuning parameters of the audit vector set based on the data distribution density and vector magnitude distribution characteristics of the audit vector set. The hierarchical graph construction module 103 is used to establish cross-layer neighbor connections for the distance relationships between vectors in the audit vector set, and to construct a hierarchical navigable small-world graph of the audit vector set by combining the temporal correlation characteristics of the audit vector set. The hierarchical parallel matching module 104 is used to receive query vectors and perform hierarchical parallel matching in a hierarchical navigable small world graph to obtain a subset of candidate audit vectors for the audit vector set. The retrieval weighted output module 105 is used to perform weighted sorting of similarity and confidence in the candidate audit vector subset, and return the audit records to the requester according to the sorting results, thereby generating a fast audit retrieval record of the original audit dataset.

[0076] In the several embodiments provided by this invention, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0077] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0078] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0079] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0080] This application embodiment can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for accelerating audit vector retrieval, characterized in that, The method includes: P1. Based on the columnar storage format, the audit records of the original audit dataset are vectorized and encoded to obtain the audit vector set of the original audit dataset; P2. Based on the data distribution density and vector magnitude distribution characteristics of the audit vector set, determine the dynamic optimization parameters of the audit vector set; P3. Establish cross-layer neighbor connections for the distance relationships between vectors in the audit vector set, and construct a hierarchical navigable small-world graph of the audit vector set by combining the temporal correlation characteristics of the audit vector set. P4. Receive the query vector and perform hierarchical parallel matching in the hierarchical navigable small world graph to obtain a subset of candidate audit vectors for the audit vector set; P5. The similarity and confidence scores of the candidate audit vector subsets are weighted and sorted, and the audit records are returned to the requester according to the sorting results, generating a fast audit retrieval record of the original audit dataset.

2. The audit vector retrieval speed-up method as described in claim 1, characterized in that, The method of vectorizing the audit records of the original audit dataset based on columnar storage format to obtain the audit vector set of the original audit dataset includes: Parse each audit record in the original audit dataset and integrate the structured field content and unstructured text content in the audit record into a field-value set of the original audit dataset; The structured field content in the field-value set is mapped to categories and values, and the mapped discrete categories are converted into integer codes to generate the first feature sub-vector of the field-value set. Word embedding vectorization is performed on the unstructured text content in the field-value set. The embedded text fragments are then converted into fixed-dimensional semantic vectors to generate the second feature sub-vector of the field-value set. Concatenate the first feature vector and the second feature vector in field order; The concatenated total vector is normalized to obtain the audit vector of the audit records; All audit vectors are written to a memory buffer in columnar storage format to form the audit vector set of the original audit dataset.

3. The audit vector retrieval speed-up method as described in claim 1, characterized in that, The step of determining the dynamic optimization parameters of the audit vector set based on the data distribution density and vector magnitude distribution characteristics of the audit vector set includes: Traverse the audit vector set, calculate the average distance between each audit vector and several of its nearest neighbors, and generate the data distribution density level of the audit vector set based on the distribution of the average distance values. The magnitudes of each audit vector in the audit vector set are arranged in order, and the minimum, maximum, and median values ​​of the magnitudes are aggregated into a magnitude distribution feature vector of the audit vector set. The data distribution density level is compared with the modulus distribution feature vector by a threshold comparison; Based on the comparison results, determine the dynamic list parameters of the audit index structure; Based on the range of the difference between the maximum and minimum values ​​in the modulus distribution feature vector, the dynamic list parameters of the audit index structure are adjusted so that the dynamic list parameters are positively correlated with the range of the difference, thus obtaining the dynamic tuning parameters of the audit vector set.

4. The audit vector retrieval speed-up method as described in claim 3, characterized in that, The step of determining the dynamic list parameters of the audit index structure based on the comparison results includes: The data distribution density level and modulus dispersion identifier contained in the comparison results are obtained. The modulus dispersion identifier is determined by the range of the difference between the maximum and minimum values ​​in the modulus distribution feature vector. When the data distribution density level is high density level and the modulus dispersion indicator is low dispersion indicator; Set the maximum number of neighbors in the index graph in the dynamic list parameters to the first value level, and set the dynamic list parameters in the index graph construction process to the second value level; When the data distribution density level is low density level or the modulus dispersion is high dispersion; Set the maximum number of neighbors parameter to the third value level and the dynamic list parameter to the fourth value level. The third value level is less than the first value level and the fourth value level is less than the second value level. Package the maximum number of neighbors parameter and the dynamic list parameter into a dynamic list parameter of the audit index structure.

5. The audit vector retrieval speed-up method as described in claim 1, characterized in that, The process of establishing cross-layer neighbor connections based on distance relationships between vectors in the audit vector set, and constructing a hierarchical navigable small-world graph of the audit vector set by combining the temporal correlation characteristics of the audit vector set, includes: The audit vector in the audit vector set is used as the current vector to be inserted, and a highest-level layer is assigned to the current vector to be inserted. Starting from the top-level entry node, the audit vector is filtered by distance between it and the existing node vectors in the highest layer to obtain candidate neighbor nodes for the audit vector set; Perform association analysis on the first specified number of candidate neighbor nodes to establish bidirectional neighbor connections for the audit vector set; Audit vectors whose timestamp difference in the audit vector set is less than a preset time period threshold are marked as a homogeneous vector group; Perform internal joins on the same source vector groups to obtain the direct join paths of the audit vector set; Using direct connection paths and audit vectors in the audit vector set as key elements, a hierarchical navigable small-world graph of the audit vector set is constructed.

6. The audit vector retrieval speed-up method as described in claim 5, characterized in that, The construction of a hierarchical, navigable small-world graph of the audit vector set, using direct connection paths and audit vectors in the audit vector set as key elements, includes: Each audit vector in the audit vector set is treated as an independent graph node, and the independent graph node includes the audit vector value and the node identifier. Bidirectional neighbor connections and direct connection paths are used as navigation graph association paths, and the identifiers and connection direction attributes of the two ends of the navigation graph association path are recorded. Based on the order in which the navigation graph associated paths are established, the independent graph nodes are associated with the navigation graph associated paths by attributes to obtain the graph structure of the audit vector set; By organizing nodes at different levels in the graph structure according to their respective levels, a cross-level bidirectional navigation structure for the audit vector set is obtained. The entire navigation graph association path between all nodes in the cross-layer bidirectional navigation structure is defined as a hierarchical navigable small-world graph of the audit vector set.

7. The audit vector retrieval speed-up method as described in claim 1, characterized in that, The received query vector is subjected to hierarchical parallel matching in a hierarchical navigable small-world graph to obtain a subset of candidate audit vectors for the audit vector set, including: Receive the query vector from the external input, use the query vector as the starting point for retrieval, and start the retrieval process from the top-level entry node of the hierarchical navigable small world graph; The process retrieval starts from the top-level entry node of the hierarchical navigable small world graph, using the query vector received from external input as the starting point. In the process retrieval, a threshold similarity test is performed on the distance between the node vectors of the parallel processing steps to be evaluated in the hierarchical navigable small world graph and the query vector to obtain candidate exit nodes of the hierarchical navigable small world graph. The candidate exit nodes are merged and deduplicated to obtain the set of valid exit nodes in the hierarchical navigable small world graph; Using each node in the effective exit node set as the entry node of the next layer, repeat the above parallel processing and filtering operations, and search downwards in sequence until the bottom layer of the hierarchical navigable small world graph is reached. At the underlying level, block-based parallel matching is performed on all nodes to collect audit vectors corresponding to nodes whose similarity distance between query vectors is less than a preset recall threshold, thus obtaining a subset of candidate audit vectors for the audit vector set.

8. The audit vector retrieval speed-up method as described in claim 7, characterized in that, The formula for calculating the distance similarity is as follows: ； In the formula, For distance similarity, For query vector, Let be the audit vector corresponding to any node in the hierarchical navigable small-world graph. To query the magnitude of the vector, For the magnitude of the audit vector, This represents the total number of fields in the audit vector. For the audit vector One field, To preset importance weight coefficients, As an indicator function, the query vector and the audit vector are in the th... When the values ​​of the fields are the same, the value is 0; when they are different, the value is 1. ∏ is the product operator.

9. The audit vector retrieval speed-up method as described in claim 1, characterized in that, The process of weighting and sorting the candidate audit vector subsets based on similarity and confidence, and then returning the audit records to the requester according to the sorting results to generate a fast audit retrieval record for the original audit dataset, includes: The distance similarity and confidence scores are weighted and fused to obtain the comprehensive ranking score of the audit vector set; Based on the comprehensive ranking score, select a specified number of candidate audit vectors from the top of the candidate audit vector subset; The extracted raw audit records are organized into a search result list according to the order in the queue, and the search result list is returned to the requester. Simultaneously, the query vector used in this retrieval, the size of the candidate audit vector subset, and the summary of the first record of the sorted candidate vector queue are saved to the audit retrieval log to generate a fast audit retrieval record of the original audit dataset.

10. An audit vector retrieval speed-up system, characterized in that, The system for implementing the audit vector retrieval speed-up method according to claim 1 includes: The audit vector encoding module is used to vectorize the audit records of the original audit dataset based on the columnar storage format to obtain the audit vector set of the original audit dataset. The dynamic parameter tuning module is used to determine the dynamic tuning parameters of the audit vector set based on the data distribution density and vector magnitude distribution characteristics of the audit vector set. The hierarchical graph construction module is used to establish cross-layer neighbor connections for the distance relationships between vectors in the audit vector set, and to construct a hierarchical navigable small-world graph of the audit vector set by combining the temporal correlation characteristics of the audit vector set. The hierarchical parallel matching module is used to receive query vectors and perform hierarchical parallel matching in a hierarchical navigable small world graph to obtain a subset of candidate audit vectors for the audit vector set. The retrieval weighted output module is used to perform weighted sorting of similarity and confidence in the candidate audit vector subset, and return the audit records to the requester according to the sorting results, generating a fast audit retrieval record of the original audit dataset.

Images