Multi-dimensional construction and retrieval method of full-scene electronic conference archives

Abstract

The application belongs to the technical field of electronic archives management, and particularly relates to a multi-dimensional construction and retrieval method of full-scene conference electronic archives, which comprises the following steps: collecting conference data and extracting multi-dimensional structured metadata, constructing initial archive records and multi-dimensional inverted indexes, performing difference coding on a document ID list, adopting storage word compression storage by dividing a selector area and a data area, and adapting overflow processing mode coding exceptions by a composite decoding descriptor containing a decoding path identifier and an exception strategy code; when retrieving, locating compressed data blocks, scheduling a vectorized decoding function, performing parallel decompression through a single instruction multiple data stream instruction set, restoring absolute document IDs through vectorized prefixes, selecting an intersection strategy in combination with global profiling data, calculating intersection sparsity to update profiling data and generating a final retrieval result. The application takes into account compression efficiency and decoding speed, optimizes a joint retrieval strategy, and improves large-scale conference archive retrieval response efficiency.

Description

Technical Field

[0001] This invention relates to the field of electronic records management technology. More specifically, this invention relates to a multi-dimensional construction and retrieval method for electronic records of meetings across all scenarios. Background Technology

[0002] With the increasing frequency of conferences and events, the electronic archives generated during these events, including audio and video recordings, text materials, and meeting documents, are growing in massive quantities. To support the need for joint retrieval under multi-dimensional conditions, existing conference archive management systems typically construct inverted indexes based on multi-dimensional information such as personnel, time, topic, and content, and generally use differential encoding to compress the document ID list in the index to reduce storage overhead.

[0003] As the size of archives continues to increase, the volume of the document ID list corresponding to the inverted index is constantly expanding, becoming a key bottleneck restricting system storage costs and I / O transmission efficiency. Unlike general web pages or regular log data, the generation of electronic archives for full-scenario conferences exhibits significant spatiotemporal intermittency and concurrent complexity. In large-scale conferences, document ID allocation is extremely dense within local time windows, but when mapping global multidimensional inverted indexes across topics and sub-forums, the distribution of document IDs shows extremely uneven jumps. This unique data distribution leads to frequent large-span abnormal differences in adjacent ID difference sequences. Although conventional difference encoding can reduce volume to some extent, to accommodate such abnormal values ​​generated by the concurrency of conferences, conventional compression algorithms are often forced to uniformly increase the storage bit width of the entire data block. This not only significantly weakens the compression effect but also adds additional computational burden to the subsequent decoding stage.

[0004] During the retrieval execution phase, traditional solutions often use a serial approach to traverse and decode the ID difference sequence one by one, which cannot fully utilize the parallel computing capabilities of modern processors. When faced with large-scale inverted lists or joint queries containing multiple search terms, this serial decoding method can easily become a performance bottleneck that restricts the response speed of the entire retrieval chain.

[0005] Furthermore, in multi-dimensional joint retrieval scenarios, the system needs to perform intersection operations on the document ID sets corresponding to multiple search terms. Existing systems typically only execute the complete processing flow for a single query, without accumulating and reusing the distribution characteristics of the document ID sets revealed during the query process. Due to the failure to effectively record and utilize this analytical data, the system cannot dynamically optimize the set traversal strategy for subsequent combinations of the same search terms. As a result, when faced with high-frequency queries of the same combination, it is still necessary to repeatedly perform inefficient sequential traversal and comparison operations, making it difficult to continuously improve retrieval efficiency.

[0006] In summary, existing technologies for indexing and retrieving electronic conference archives have limitations in areas such as anomaly compression processing, parallel decoding efficiency, and joint retrieval strategy optimization, making them unsuitable for high-concurrency retrieval needs under large-scale archival data. There is an urgent need for a solution for indexing and retrieving electronic conference archives that can simultaneously achieve index compression efficiency, high-speed decoding performance, and optimized joint retrieval strategies. Summary of the Invention

[0007] To address the problems of high storage overhead, low decoding efficiency, and slow response speed of existing electronic meeting archive indexes, which make it difficult to adapt to the high-concurrency retrieval needs of large-scale archive data, this invention provides a multi-dimensional construction and retrieval method for electronic meeting archives across all scenarios.

[0008] A multi-dimensional construction and retrieval method for electronic archives of meetings across all scenarios includes: S1. Extracting structured metadata based on meeting data to construct initial archive records and a multi-dimensional inverted index; subtracting adjacent items from the document ID list in the multi-dimensional inverted index to obtain an ID difference sequence; identifying differences exceeding a preset bit width as abnormal differences; compressing and storing the ID difference sequence using storage words divided into selector and data areas to form a compressed document ID list data block; wherein, the selector area is mapped with a composite decoding descriptor containing a decoding path identifier and an anomaly policy code; and encoding the abnormal differences according to the corresponding overflow processing mode selected based on the anomaly policy code; S2. Receiving retrieval requests and extracting a set of search terms, locating each The system retrieves a list of compressed document IDs corresponding to the search terms; reads the stored words to obtain the composite decoding descriptor; calls the corresponding vectorized decoding function based on the decoding path identifier and the exception policy code; decompresses the ID difference sequence in parallel using a single instruction multiple data stream instruction set; and restores it to an absolute document ID set using a vectorized prefix sum operation; S3, determines the intersection strategy based on the global profiling data in the system, and performs an intersection operation on the absolute document ID set based on the intersection strategy to obtain the intersection absolute document ID set; determines the intersection sparsity corresponding to the search term set based on the intersection result and updates the global profiling data; and generates a search result by combining the intersection absolute document ID set with the initial archive record.

[0009] This invention first extracts structured metadata from conference data, constructs a multi-dimensional inverted index, and compresses and stores the document ID list using difference encoding combined with partitioned storage words. A differentiated overflow handling mode is set up for abnormal differences, ensuring compression efficiency while being compatible with uneven document ID distribution. During the retrieval phase, the corresponding vectorized decoding function is scheduled based on a composite decoding descriptor. The difference sequence is decompressed in parallel using a single-instruction, multi-data-stream instruction, and the absolute document ID is quickly restored using vectorized prefix sums. During joint retrieval, the global data analysis selection intersection strategy is reused, and the intersection sparsity is updated to optimize subsequent queries. This solution effectively reduces index storage overhead, improves decoding and joint retrieval efficiency, and adapts to the high-concurrency retrieval needs of large-scale conference archives.

[0010] Preferably, the process of subtracting adjacent items from the document ID list in the multidimensional inverted index to obtain the ID difference sequence includes: initializing the previous document ID to a preset initial value, and reading the current document ID sequentially according to the sorting order of the document ID list;

[0011] The ID difference is obtained by subtracting the previous document ID from the current document ID, and the previous document ID is updated to the current document ID. This process is repeated to traverse the document ID list and generate the ID difference sequence.

[0012] Preferably, the abnormal difference is encoded by selecting the corresponding overflow processing mode according to the abnormal policy code, including: when the abnormal policy code indicates the in-situ escape mode, the position of a data item in the current storage word is used as an escape mark, the adjacent bit space after merging the escape marks is used to store a single abnormal difference, and the remaining data items of the current storage word are used to store non-abnormal differences according to the normal bit width. When the exception policy code indicates the additional block mode, the data area of ​​the current storage word is used to indicate that at least one subsequent storage word constitutes an additional block, and the exception difference value corresponding to the current storage word is centrally stored in the additional block.

[0013] By setting two overflow handling modes, namely in-situ escape and additional block, a differentiated encoding method is adopted for different abnormal difference scenarios. This allows for compatibility with large numerical abnormal differences without uniformly increasing the data block storage bit width, while taking into account both the compression efficiency of regular differences and the storage adaptability of abnormal differences.

[0014] Preferably, when the exception policy code indicates the additional block mode, the data area of ​​the current storage word is used to indicate that at least one subsequent storage word constitutes an additional block, and the exception difference value corresponding to the current storage word is centrally stored in the additional block, including: using the data area of ​​the current storage word as an additional block control field, and the additional block control field recording the exception difference value location bitmap, the number of ordinary data storage words and the number of storage words of the additional block according to a preset field layout; The ID difference sequence is divided into multiple encoding batches; the non-abnormal differences in the current encoding batch are written into the ordinary data storage word immediately following the additional block control field in the order of their positions in the ID difference sequence; and the abnormal differences in the current encoding batch are written into the subsequent additional blocks in the order of their positions in the ID difference sequence.

[0015] By setting an additional block control field in the data area to record the position bitmap and the number of stored words, normal differences and abnormal differences are stored in separate partitions. During decoding, the sequence order can be quickly restored based on the bitmap, adapting to scenarios with multiple abnormal differences or large bit width abnormal differences, thus improving encoding flexibility and space utilization.

[0016] Preferably, the step of restoring the absolute document ID set through vectorized prefix sum operation includes: dividing the decompressed ID difference sequence into multiple vector calculation batches, and performing the following steps for each vector calculation batch: loading an initial vector containing multiple decompressed ID differences; shifting the elements in the initial vector to a specified position in the direction of increasing element index and padding the empty low index channels with zeros; adding the shifted vector to the initial vector element by element for at least one round of local accumulation to obtain a local prefix sum vector; determining the reference scalar corresponding to the current vector calculation batch; constructing a broadcast vector where each element is the reference scalar; adding the local prefix sum vector to the broadcast vector element by element to obtain the absolute document ID set of the current vector calculation batch; and summing the absolute document ID sets of all vector calculation batches to obtain the absolute document ID set.

[0017] Preferably, the parallel decompression of the ID difference sequence using a single instruction multiple data stream instruction set includes: dividing the ID difference sequence into multiple encoding batches according to a preset rule; When the exception policy code indicates the in-situ escape mode, the escape flag in the current storage word is identified by the single instruction multiple data stream instruction set, and the data in the extended bit space adjacent to the escape flag is read as an exception difference output. When the exception policy code indicates the append block mode, the data area of ​​the current storage word is parsed to obtain the bitmap of the exception difference position in the subsequent append block and the append block length. The non-exception difference and the exception difference in the append block are extracted respectively, and the exception difference is inserted into the corresponding position according to the exception difference position bitmap to restore the complete ID difference sequence of the current encoding batch.

[0018] By matching corresponding parallel decompression logic to the two exception handling modes, the escape flag is identified and the extended space data is read in the in-situ escape mode, and the sequence order is restored according to the bitmap in the additional block mode. Combined with the single instruction multiple data stream instruction, batch differential parallel decompression is achieved, thereby improving the decoding execution efficiency.

[0019] Preferably, determining the intersection sparsity of the search term set based on the intersection result includes: calculating the number of common intersection elements of the absolute document ID sets corresponding to each search term; Calculate the number of elements in the union of the absolute document ID sets corresponding to each search term; The intersection-union ratio is obtained by dividing the number of elements in the common intersection by the number of elements in the union. The difference between the value 1 and the intersection-union ratio is determined as the intersection sparsity corresponding to the set of search terms.

[0020] Preferably, the global profile data includes historical intersection sparsity smoothing values; determining the intersection strategy based on the global profile data in the system includes: when the historical intersection sparsity smoothing value is less than a first threshold, determining to perform intersection using a sequential traversal method; when the historical intersection sparsity smoothing value is greater than or equal to the first threshold and less than a second threshold, determining to perform intersection using a jump search method; when the historical intersection sparsity smoothing value is greater than or equal to the second threshold, determining to perform intersection using a block-level jump traversal method. The process of updating the global profile data includes: performing a weighted calculation on the intersection sparsity and the current historical intersection sparsity smoothing value recorded in the global profile data to obtain the updated historical intersection sparsity smoothing value and storing it in the global profile data.

[0021] Preferably, the extraction of structured metadata based on meeting data includes: extracting text, timestamp records, and speaker identity information from the collected multidimensional audiovisual data stream; and mapping the extracted text, the timestamp records, and the speaker identity information as feature keys of different dimensions to the multidimensional inverted index.

[0022] By extracting multi-dimensional structured metadata such as text, timestamps, and speaker identities and deeply mapping them to a multi-dimensional inverted index, the index dimensions of the archives are enriched, supporting multi-dimensional retrieval needs and improving the information structuring level of the conference archives.

[0023] Preferably, the extraction of structured metadata based on meeting data further includes: extracting specified business entities and meeting topics from the text; correspondingly, the construction of initial archive records includes: combining the meeting topics, the text, the timestamp records, the speaker's identity information, and the specified business entities to construct the initial archive records, and establishing a multi-dimensional search entry for the document ID list based on the specified business entities and meeting topics.

[0024] By extracting business entities and meeting topics and establishing multi-dimensional search entry points, the structured metadata dimensions are further supplemented, enabling the initial archive records to contain more comprehensive meeting feature information, expanding search entry points, and improving the accuracy and coverage of multi-dimensional searches.

[0025] The technical solution of the present invention has the following beneficial technical effects: This invention addresses the multidimensional inverted indexing of electronic conference archives. It converts a list of document IDs into a difference sequence and combines a partitioned storage word structure with a composite decoding descriptor. Regular and abnormal differences are handled using standard bit-width storage and overflow processing modes, respectively. This design deeply aligns with the unique distribution pattern of conference archive IDs, characterized by local density and global jumps. By accurately separating local low-bit-width and large-span abnormal differences, it effectively addresses the uneven ID distribution problem caused by multiple concurrent topics in large conferences, ensuring compression efficiency while maintaining compatibility with uneven document ID distribution. During the retrieval phase, the composite decoding descriptor schedules the corresponding vectorized decoding function. Parallel decompression of the difference sequence is achieved using single-instruction multi-data-stream instructions, and vectorized prefix sum operations restore the differences to absolute document IDs, effectively reducing computational latency caused by serial decoding. During joint retrieval, the intersection sparsity is calculated and global analysis data is updated, allowing subsequent combinations of the same search terms to select appropriate intersection methods based on historical distribution characteristics, reducing invalid comparisons during set traversal. The above methods, when used in combination, can effectively control index storage overhead, improve the processing efficiency of decoding and intersection steps, enhance retrieval response capabilities under large-scale archival data, and adapt to high-concurrency meeting archive retrieval scenarios. Attached Figure Description

[0026] Figure 1 A flowchart illustrating the multi-dimensional construction and retrieval methods for electronic archives of meetings across all scenarios; Figure 2 This is a bar chart showing the numerical distribution of ID difference values. Detailed Implementation

[0027] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0028] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0029] This invention discloses a multi-dimensional construction and retrieval method for electronic meeting archives across all scenarios, referring to... Figure 1 This includes the following steps: S1. Extract structured metadata and build a compressed multidimensional inverted index.

[0030] This step first collects meeting data and extracts structured metadata to build initial archive records. Then, it builds a multidimensional inverted index based on the initial archive records. Finally, it performs differential encoding and compression storage on the document ID list in the index to form a compressed document ID list data block.

[0031] Provided that authorization from the data provider is obtained and relevant data management regulations are met, meeting texts, audio and video materials, and related meeting documents are acquired through methods such as the meeting management system interface, authorized data interface, or local file import. Structured metadata is then extracted from the collected multi-dimensional audiovisual data streams.

[0032] Text, timestamp records, and speaker identification information are extracted from the collected multidimensional audiovisual data stream. Specifically, audio tracks can be separated using conventional audio and video processing frameworks in this field, and pre-trained speech recognition technology can be used to convert the speech sequence into text, while extracting the corresponding start and end time points to generate timestamp records. Speaker identification information is obtained through speaker separation and voiceprint matching techniques. The extracted text, timestamp records, and speaker identification information are then used as feature keys of different dimensions and mapped to a multidimensional inverted index.

[0033] Furthermore, conventional entity recognition techniques and topic extraction algorithms in this field are employed to extract specified business entities and meeting topics from the text. Accordingly, the meeting topics, text, timestamp records, speaker identification information, and specified business entities are combined, serialized into structured data in JSON format, and an initial archive record is constructed and written to a persistent storage device. A multi-dimensional retrieval entry point is established for the document ID list based on the specified business entities and meeting topics.

[0034] A multi-dimensional inverted index is constructed based on initial archival records. Each initial archival record is assigned a globally unique document ID, which is then associated with and stored in conjunction with structured metadata, the storage path of the original meeting documents, or a Uniform Resource Locator (URL). Using attendees, meeting time, and meeting topic as index dimensions, the Ansj Chinese word segmenter, combined with a pre-defined user dictionary, stop word list, and term normalization rules, processes the structured metadata to obtain corresponding feature keys. These feature keys are then mapped to a list of document IDs containing those feature keys, arranged in ascending numerical order, forming the multi-dimensional inverted index.

[0035] The document ID list in the multidimensional inverted index is subtracted from adjacent items to obtain an ID difference sequence. Specifically, an input buffer of preferably 4KB to 8KB is set, and a batch of 32-bit unsigned integer absolute document ID sequences are loaded, with the preceding document ID variable initialized to 0. The current document ID is read sequentially according to the ascending order of the document ID list, and the preceding document ID is subtracted from the current document ID to obtain the corresponding ID difference. The preceding document ID is then updated to the current document ID. After traversing the complete document ID list, an ID difference sequence consisting of all ID differences is generated. This difference encoding process can be represented by the following relation:

[0036] In the formula, Indicates the document ID in the list of document IDs An absolute document ID, Indicates the number immediately preceding it. An absolute document ID, Indicates the first The ID difference; the preceding document ID corresponding to the first document ID is 0, therefore the first ID difference This relation is built upon the monotonically increasing characteristic of an ordered document ID sequence. By subtracting adjacent terms to remove redundant high-order bits, it converts large absolute IDs into small relative differences, providing a data foundation for subsequent compact storage with fewer bits. For example, a sequence of absolute document IDs 1000, 1005, 1006, 1010, and 1025, after difference encoding, becomes 1000, 5, 1, 4, and 15. It can be seen that most of the differences are small integers.

[0037] After obtaining the ID difference sequence, the ID difference sequence is compressed and stored using a storage word divided into a selector area and a data area, forming a compressed document ID list data block. This embodiment uses a 32-bit storage word, with the high 4 bits divided into a selector area and the low 28 bits into a data area. The 4-bit selector can support up to 16 encoding modes, sufficient to cover various scenarios from extremely small differences to large-span abnormal differences; the 28-bit data area provides enough space to accommodate multiple compactly encoded ID differences in a single storage word. The ID difference sequence is scanned, and differences whose values ​​exceed the range of normal differences that can be represented by a preset normal bit width are identified as abnormal differences; the maximum value of the normal bit width is reserved as an escape flag, therefore the range of normal differences that can be represented is 0 to... ( (For standard bit width), the value is greater than or equal to All differences were judged as abnormal differences.

[0038] The selector area maps to a composite decoder descriptor containing a decoder path identifier and an exception policy code. The composite decoder descriptor statically records information such as the decoder path, exception handling method, data item width, and number of standard data items. Dynamic control information, such as the actual number of valid data items, the location identifier of the exception difference, and the length of the append block, is recorded in the block header of the compressed data block, the data area of ​​the storage word, or the append block control field. During encoding, the corresponding overflow handling mode is selected based on the exception policy code to encode the exception difference. Overflow handling modes include in-situ escape mode and append block mode.

[0039] When the exception policy code indicates in-situ escape mode, one data item position within the current storage word is used as the escape marker. The adjacent bit spaces after merging the escape markers store a single exception difference. The remaining data items in the current storage word store non-exception differences with the normal bit width. In 4-bit wide encoding mode, the value 15 is reserved as the escape marker. The range of values ​​that can be directly encoded for ordinary ID differences is 0 to 14. When an exception difference occurs, the escape marker 15 is written to the preset data item position. The three adjacent 4-bit spaces are merged into a 12-bit extended storage area, where the binary data of the exception difference is written. The maximum representation range of the 12-bit extended storage area is 0 to 4095, which can cover most exception differences that may occur in real-world scenarios. Its bit width selection is the result of balancing storage overhead and representation range. If the exception difference itself is exactly equal to 15, it is also encoded by the extended storage area to distinguish it from the escape marker. If the exception difference exceeds the representation range of the extended storage area or the remaining space is insufficient to form a 12-bit extended storage area, the mode switches to additional block mode.

[0040] When the exception policy code indicates the supplementary block mode, the data area of ​​the current storage word is used to indicate that at least one subsequent storage word constitutes a supplementary block. The exception difference value corresponding to the current storage word is stored centrally in the supplementary block. Specifically, the data area of ​​the current storage word is used as the supplementary block control field. The supplementary block control field records the exception difference location bitmap, the number of ordinary data storage words, and the number of storage words in the supplementary block according to a preset field layout.

[0041] Before encoding, the ID difference sequence is divided into multiple encoding batches according to preset rules. In this embodiment, each encoding batch contains a maximum of 8 ID differences. This value is based on the statistical analysis of the frequency of abnormal differences in the ID list of typical documents. Grouping 8 together can maintain high encoding efficiency while controlling the overhead of the additional block field.

[0042] In a preferred embodiment, bits 0 to 7 of the additional block control field are a bitmap of abnormal difference locations. A value of 1 indicates that the current batch contains the [number] [item]. The first position represents the abnormal difference; positions 8 to 11 record the number of ordinary data storage words, with 4 bits representing 0 to 15 storage words; positions 12 to 15 record the number of additional storage words; positions 16 to 19 record the number of valid ID differences in the current batch, with 4 bits representing 0 to 15, sufficient to cover the requirement of a maximum of 8 valid differences per batch; positions 20 to 27 are reserved fields for future function expansion. The bit width settings of each field are derived based on the batch upper limit of 8, ensuring that the control information itself does not become a storage burden.

[0043] During encoding, non-abnormal differences in the current encoding batch are written to the ordinary data storage word immediately following the appendix control field, according to their order in the ID difference sequence. Abnormal differences in the current encoding batch are written to subsequent appendix blocks according to their position in the ID difference sequence. During decoding, the decoder first extracts ordinary and abnormal differences based on the bitmap and length information in the appendix control field, and then inserts the abnormal differences into the correct positions according to the bitmap, thereby completely restoring the ID difference sequence of the current encoding batch. Figure 2 The statistical distribution of ID difference values ​​in the actual meeting archive index is shown. It can be seen that the vast majority of differences are concentrated in the low bit range, and only a small number of differences that exceed the normal bit width are identified as abnormal differences. By processing them separately in the above two modes, we can avoid increasing the overall storage bit width to accommodate a few large values, which helps to improve the overall compression efficiency.

[0044] After the multidimensional inverted index and compressed document ID list data blocks are generated, the document IDs and corresponding file records are synchronously written to the Elasticsearch search engine server for subsequent extraction of file content based on document IDs. Simultaneously, the compressed document ID list data blocks are written to the inverted index data file, recording the corresponding file identifier, logical offset address, data block length, and compression format identifier. Using a composite key formed by concatenating the search term hash value and the normalized search term as the key, the above location information is written as the value to the key-value database, thereby establishing the association between the initial file records, the multidimensional inverted index, the compressed data blocks, and the forward index file records.

[0045] S2. Decompress the vectorized data and restore the absolute document ID set.

[0046] The system receives retrieval requests and parses the corresponding set of search terms. Based on the set of search terms, it locates the compressed document ID list data block corresponding to each search term. It reads the storage word to obtain the composite decoding descriptor, calls the corresponding vectorized decoding function based on the decoding path identifier and the exception policy code, decompresses the ID difference sequence in parallel through a single instruction multiple data stream instruction set, and restores it to the absolute document ID set through vectorized prefix sum operation.

[0047] Specifically, the search text input by the user is received based on the Spring Boot framework. HanLP is used for dependency parsing and part-of-speech tagging to filter stop words and extract core content words. Then, the same word segmenter, user dictionary, stop word list and term normalization rules as in the index building stage are used for processing to obtain the set of search terms corresponding to the feature keys in the multidimensional inverted index.

[0048] For each canonical search term, a 64-bit hash value is calculated using the MurmurHash3 algorithm. This hash value is then concatenated with the canonical search term to form a composite key, which is used to query the key-value database. Upon successful query, the corresponding file identifier, logical offset address, data block length, and compression format identifier are read. Based on the location information, the compressed document ID list data block is loaded into the memory buffer. If no corresponding data block is found for a search term, the set of absolute document IDs corresponding to that search term is set to an empty set. For join requests using logical AND relations, an empty search result can be returned directly.

[0049] Based on the compression format identifier and data block header, identify the ordinary data storage word, the additional block control storage word, and the additional storage word. For the ordinary data storage word and the additional block control storage word, extract the high 4 bits of the selector value, and use the selector value to query the static hash table to obtain the corresponding composite decoding descriptor; for the additional storage word, directly read the 32-bit abnormal difference value according to the additional block length recorded in the additional block control field, without extracting the selector value.

[0050] During system initialization, a function mapping table is established based on different combinations of decoding path identifiers and exception policy codes. For each combination, a corresponding vectorized decoding function, shift sequence, and mask are configured. During decoding, the decoding path identifier and exception policy code are combined into a function index. The corresponding vectorized decoding function is called based on the function index, and the ID difference sequence is decompressed in parallel using a single-instruction multiple-data-stream instruction set. The parallel decompression process divides the ID difference sequence into multiple encoding batches according to preset rules, and executes corresponding decoding logic for different exception handling modes.

[0051] When the exception policy code indicates in-situ escape mode, the escape flag within the current memory word is identified using the Single Instruction Multiple Data (SIMD) instruction set. Data from the extended bit space adjacent to this escape flag is then read and output as an exception difference. Other ordinary differences are extracted in parallel using a shift sequence and a mask. For example, in 4-bit wide encoding mode, shift sequences 0, 4, 8, 12, 16, 20, 24 and a mask of 0x0F are used to extract all ordinary differences in parallel through shift and bitwise AND operations. These shift values ​​and masks correspond one-to-one with the bit width of each data item, ensuring that all ordinary differences can be extracted in a single SIMD operation.

[0052] When the exception policy code indicates the appended block mode, the data area of ​​the current storage word is parsed to obtain the bitmap of exception difference positions in subsequent appended blocks and the appended block length. Non-exception differences in the ordinary data storage word and exception differences in the appended block are extracted separately, and the exception differences are inserted into their corresponding positions according to the exception difference position bitmap, thus restoring the complete ID difference sequence of the current encoding batch. The encoding batch division used during decoding remains consistent with that used during encoding; each batch corresponds to a storage word and the ID difference group decompressed from its possible associated appended blocks.

[0053] After decompression to obtain the complete ID difference sequence, it is restored to the absolute document ID set through vectorized prefix sum operation. Specifically, the decompressed ID difference sequence is divided into multiple vector calculation batches, and the following steps are performed for each vector calculation batch.

[0054] An initial vector containing multiple decompressed ID differences is loaded. In this embodiment, four 32-bit ID differences are loaded into a 128-bit vector register at a time. The 128-bit width matches the vector register width of current mainstream processors' SSE, NEON, and other single-instruction multiple-data instruction sets. Loading four 32-bit integers at a time fully utilizes the register bandwidth. Elements in the initial vector are shifted by a specified number of positions in the direction of increasing element index, and zeros are padded in the vacated low-index channels. The shifted vector is then added element-wise to the initial vector, performing at least one round of local accumulation to obtain a local prefix sum vector.

[0055] This embodiment employs a two-round shift-and-add operation: In the first round, the initial vector is shifted one element position (32 bits) in the direction of increasing element index, padded with zeros, and then added element-by-element to the initial vector to obtain the first local accumulation vector; in the second round, the first local accumulation vector is shifted two element positions (64 bits) in the direction of increasing element index, padded with zeros, and then added element-by-element to the first local accumulation vector to obtain the local prefix sum vector. The reason for using a two-round shift-and-add operation in the vectorized prefix sum algorithm is that, for a vector of 4 elements, the first round of shifting one position and accumulating allows each element to obtain the accumulated value of its immediately preceding element; the second round of shifting two positions and accumulating allows each element to obtain the accumulated values ​​of the previous two elements, thus completing the entire prefix sum calculation within log24 equal to 2 rounds of operations. The time complexity is reduced from O(n) in the serial operation to O(log2n), effectively reducing the data dependency formed by the element-by-element serial accumulation.

[0056] Determine the baseline scalar for the current vector computation batch: if the current batch is the first batch, set the baseline scalar to 0; if the current batch is not the first batch, use the last absolute document ID obtained from the previous batch as the baseline scalar. Construct a broadcast vector where each element is the baseline scalar. Add the local prefix sum vector to the broadcast vector element by element to obtain the set of absolute document IDs for the current batch. If the batch ends with fewer than four valid differences, pad the empty channels with zeros and output only the document IDs corresponding to the valid channels.

[0057] Summarize the absolute document IDs of all vector calculation batches to obtain the complete absolute document ID set corresponding to the search term.

[0058] To facilitate understanding, the prefix sum reconstruction process is illustrated using a batch of non-first vectors as an example: Assume the last absolute document ID obtained from the previous batch decoding is 1000, and the ID differences for the current batch are 5, 1, 4, and 15 respectively. The initial vector [5,1,4,15] is loaded, and after the first round of shifting and accumulation, the first local accumulation vector [5,6,5,19] is obtained. After the second round of shifting and accumulation, the local prefix sum vector [5,6,10,25] is obtained. A broadcast vector is constructed using 1000 as the base scalar, and after element-wise addition with the local prefix sum vector, the absolute document ID sequence for the current batch is obtained as 1005, 1006, 1010, and 1025.

[0059] S3. Perform adaptive intersection operation and generate search results.

[0060] The intersection strategy is determined based on the global analysis data within the system, and the intersection operation is performed on the absolute document ID set based on the intersection strategy to obtain the intersection absolute document ID set; the intersection sparsity corresponding to the search term set is determined based on the intersection result and the global analysis data is updated; the search results are generated by combining the intersection absolute document ID set with the initial file records.

[0061] During system initialization, global profile data is maintained in the local process space. This global profile data includes historical intersection sparsity smoothing values. This embodiment uses a hash mapping structure that supports concurrent access to store the global profile data. The key is a string of search terms concatenated in lexicographical order, and the value is the historical intersection sparsity smoothing value. This hash mapping structure supports secure read and write operations in high-concurrency environments and can adapt to scenarios where multiple users search simultaneously.

[0062] Before performing the set intersection operation, first check if there is an entry in the global analysis data corresponding to the current search term combination. If there is no corresponding entry, the intersection is performed by default using a sequential traversal method; if there is a corresponding entry, the historical intersection sparsity smoothing value is read, and the intersection strategy is determined based on the comparison result with the preset threshold.

[0063] In this embodiment, the first threshold is set to 0.5 and the second threshold to 0.8. The values ​​of these two thresholds are based on the following: the theoretical range of intersection sparsity is 0 to 1, where 0 indicates complete overlap of the sets and 1 indicates no overlap. Statistical analysis of a large number of historical query logs reveals that when the intersection sparsity is below 0.5, the overlap of the document ID sets is high, and there are many common elements between the sets. Sequential traversal can hit the intersection elements with a high probability, and the additional benefit from skipping operations is limited. When the intersection sparsity is between 0.5 and 0.8, the dispersion of the sets is moderate, and jump search can leverage its advantage of rapid progress when there are large gaps. When the intersection sparsity reaches above 0.8, the overlapping elements between the sets are scarce, the document IDs are sparsely distributed, and block-level jump traversal can significantly reduce the number of invalid comparisons by skipping the entire data block at once. In actual deployment, the above thresholds can be fine-tuned by comparing the average response time under different threshold combinations through A / B testing based on the query performance monitoring data accumulated after the system goes online, and the boundary value that makes the overall system performance better can be selected.

[0064] The specific execution methods of each intersection strategy are as follows: When the sparsity smoothing value of the historical intersection is less than the first threshold of 0.5, the sequential traversal method is used to perform the intersection. The sequential traversal method involves simultaneously traversing the two ordered document ID lists, comparing the document IDs pointed to by the current pointers one by one. If the values ​​are equal, they are recorded as intersection elements and both pointers are moved simultaneously; otherwise, the pointer pointing to the smaller document ID is moved.

[0065] When the historical intersection sparsity smoothing value is greater than or equal to the first threshold of 0.5 and less than the second threshold of 0.8, the skip search method is used to perform intersection finding. The skip search method utilizes the ordered nature of the document ID list. Instead of stepping one element when moving the pointer, it calculates the jump step size based on the difference between the document IDs pointed to by the two pointers, thereby quickly approaching the possible matching position and reducing the number of single-step comparisons.

[0066] When the historical intersection sparsity smoothing value is greater than or equal to the second threshold of 0.8, the block-level skip traversal method is used to perform intersection calculation. The block-level skip traversal method pre-divides the ordered set of absolute document IDs into multiple consecutive data blocks and records the first and last document IDs of each data block. When calculating the intersection, the current document ID to be matched is compared with the last document ID of each data block. If the last document ID is still less than the document ID to be matched, then all document IDs in that data block cannot be matched, and the block is skipped directly to the next block. Only when the document ID to be matched falls within the first and last interval of a block is the block entered to perform a fine comparison at the document ID level.

[0067] When there are more than two search terms, the intersection is iteratively calculated in ascending order of the number of elements in each absolute document ID set. That is, the intersection of the two smallest sets is calculated first, then the intermediate result is intersected with the next smaller set, and so on. This sorting strategy can reduce the number of candidate elements as much as possible in the earlier intersection rounds, reducing the computational load in subsequent rounds.

[0068] After the intersection is completed, the sparsity of the intersection corresponding to the set of search terms is determined based on the intersection result. The calculation process is as follows.

[0069] Calculate the number of common intersection elements in the absolute document ID sets corresponding to each search term, and then calculate the number of union elements in the absolute document ID sets corresponding to each search term. Divide the number of common intersection elements by the number of union elements to obtain the intersection-union ratio. Based on the difference between the value 1 and the intersection-union ratio, determine the intersection sparsity corresponding to the search term sets. This calculation process can be represented by the following relation:

[0070] In the formula, This indicates the number of search terms participating in the combined search. Indicates the first The set of absolute document IDs corresponding to each search term Denotes the common intersection of all sets. Denotes the union of all sets. Indicates the number of elements in the set. This indicates the sparsity of the intersection. The relationship is constructed based on the intersection-union ratio (IUR) index of set similarity. By taking the complement value of the IUR, it characterizes the degree of dispersion among multiple document ID sets. The closer the value is to 1, the lower the overlap between sets. This is used as the basis for judging the choice of intersection strategy.

[0071] When the union of all sets is empty, the denominator of the formula is zero and cannot be calculated. In this case, an empty search result is returned directly, without calculating the intersection sparsity or updating the global analysis data.

[0072] To facilitate understanding, let's illustrate sparsity calculation using a scenario with two search terms: Suppose the set of absolute document IDs corresponding to search term A is {1005, 1006, 1010, 1025}, and the set of absolute document IDs corresponding to search term B is {1006, 1010, 1050}. Using a two-pointer synchronous traversal, we find the common elements 1006 and 1010. The number of elements in the common intersection is 2. The number of elements in the union is calculated by adding the total number of elements in set A (4) to the total number of elements in set B (3), and then subtracting the number of elements in the intersection (2), resulting in 5. The intersection-union ratio is 2 divided by 5, which equals 0.4. The corresponding intersection sparsity S is 1 minus 0.4, which equals 0.6.

[0073] After obtaining the current intersection sparsity, update the global profile data. The intersection sparsity is weighted and calculated with the smoothed historical intersection sparsity values ​​recorded in the global profile data to obtain updated smoothed historical intersection sparsity values, which are then stored in the global profile data. The update process can be represented by the following formula:

[0074] In the formula, This represents the sparsity smoothing value of the historical intersections before the update. This indicates the sparsity of the intersection obtained in this calculation. As a smoothing coefficient, in this embodiment Take 0.8, This represents the updated historical intersection sparsity smoothing value. This formula is built upon the concept of an exponentially weighted moving average, with historical values ​​accounting for 80% of the weight. This maintains the stability of the strategy selection and avoids frequent switching of the intersection method due to occasional fluctuations in a single query; the current value accounts for 20% of the weight, allowing the system to gradually track and adapt to slow, trending changes in data distribution. Smoothing coefficient. The value can be tested by using a grid search method on the validation query set to determine different values. The value is determined by the relationship between the strategy switching frequency and the average response time. Generally speaking, for scenarios with high stability requirements, a value of 0.8 to 0.9 can be used, while for scenarios with high sensitivity requirements, a value of 0.6 to 0.7 can be used.

[0075] If there is no corresponding historical record for the current combination of search terms, the current intersection sparsity will be directly written into the global analysis data as the initial historical intersection sparsity smoothing value.

[0076] Finally, the absolute document IDs of the intersection obtained by intersection are sent to the Elasticsearch server as query conditions to query the corresponding initial archive records, extract the context fragments matching the search terms and the location information of the original meeting documents, and encapsulate the document IDs, context fragments, and source file location information into search results, which are then fed back to the user terminal.

[0077] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multi-dimensional construction and retrieval method for electronic archives of meetings across all scenarios, characterized in that: include: S1. Extract structured metadata from meeting data to build initial archive records and a multidimensional inverted index; The document ID list in the multidimensional inverted index is subtracted from adjacent items to obtain an ID difference sequence. Differences exceeding a preset bit width are identified as abnormal differences. The ID difference sequence is compressed and stored using a storage word divided into a selector area and a data area to form a compressed document ID list data block. The selector area is mapped to a composite decoding descriptor containing a decoding path identifier and an exception policy code. The abnormal difference is encoded using the corresponding overflow handling mode selected according to the exception policy code. S2: A search request is received and a set of search terms is extracted. The compressed document ID list data block corresponding to each search term is located. The storage word is read to obtain... The composite decoding descriptor calls the corresponding vectorized decoding function based on the decoding path identifier and the exception policy code, decompresses the ID difference sequence in parallel using a single instruction multiple data stream instruction set, and restores it to an absolute document ID set through a vectorized prefix sum operation; S3, determines the intersection strategy based on the global analysis data in the system, and performs an intersection operation on the absolute document ID set based on the intersection strategy to obtain the intersection absolute document ID set; determines the intersection sparsity corresponding to the search term set based on the intersection result and updates the global analysis data, and generates search results by combining the intersection absolute document ID set with the initial file record.

2. The method for multi-dimensional construction and retrieval of electronic meeting archives across all scenarios according to claim 1, characterized in that, The process of subtracting adjacent items from the document ID list in the multidimensional inverted index to obtain the ID difference sequence includes: initializing the previous document ID to a preset initial value, and reading the current document ID sequentially according to the sorting order of the document ID list; The ID difference is obtained by subtracting the previous document ID from the current document ID, and the previous document ID is updated to the current document ID. This process is repeated to traverse the document ID list and generate the ID difference sequence.

3. The method for multi-dimensional construction and retrieval of electronic meeting archives across all scenarios according to claim 1, characterized in that, The abnormal difference is encoded by selecting the corresponding overflow handling mode according to the abnormal policy code, including: when the abnormal policy code indicates the in-situ escape mode, the position of a data item in the current storage word is used as the escape mark, the adjacent bit space after merging the escape mark is used to store a single abnormal difference, and the remaining data items of the current storage word are used to store non-abnormal differences according to the normal bit width. When the exception policy code indicates the additional block mode, the data area of ​​the current storage word is used to indicate that at least one subsequent storage word constitutes an additional block, and the exception difference value corresponding to the current storage word is centrally stored in the additional block.

4. The method for multi-dimensional construction and retrieval of electronic meeting archives across all scenarios according to claim 3, characterized in that, When the exception policy code indicates the additional block mode, the data area of ​​the current storage word is used to indicate that at least one subsequent storage word constitutes an additional block. The exception difference corresponding to the current storage word is centrally stored in the additional block, including: using the data area of ​​the current storage word as an additional block control field. The additional block control field records the exception difference location bitmap, the number of ordinary data storage words, and the number of storage words of the additional block according to a preset field layout. The ID difference sequence is divided into multiple encoding batches; the non-abnormal differences in the current encoding batch are written into the ordinary data storage word immediately following the additional block control field in the order of their positions in the ID difference sequence; and the abnormal differences in the current encoding batch are written into the subsequent additional blocks in the order of their positions in the ID difference sequence.

5. The method for multi-dimensional construction and retrieval of electronic meeting archives across all scenarios according to claim 1, characterized in that, The process of restoring the absolute document ID set through vectorized prefix sum operation includes: dividing the decompressed ID difference sequence into multiple vector calculation batches, and performing the following steps for each vector calculation batch: loading an initial vector containing multiple decompressed ID differences; shifting the elements in the initial vector to a specified position in the direction of increasing element index and padding the empty low index channels with zeros; adding the shifted vector element-wise to the initial vector for at least one round of local accumulation to obtain a local prefix sum vector; determining the reference scalar corresponding to the current vector calculation batch; constructing a broadcast vector where each element is the reference scalar; adding the local prefix sum vector element-wise to the broadcast vector to obtain the absolute document ID set of the current vector calculation batch; and summing the absolute document ID sets of all vector calculation batches to obtain the absolute document ID set.

6. The method for multi-dimensional construction and retrieval of electronic meeting archives across all scenarios according to claim 1, characterized in that, The method of parallel decompressing the ID difference sequence using a single instruction multiple data stream instruction set includes: dividing the ID difference sequence into multiple encoding batches according to a preset rule; When the exception policy code indicates the in-situ escape mode, the escape flag in the current storage word is identified by the single instruction multiple data stream instruction set, and the data in the extended bit space adjacent to the escape flag is read as an exception difference output. When the exception policy code indicates the append block mode, the data area of ​​the current storage word is parsed to obtain the bitmap of the exception difference position in the subsequent append block and the append block length. The non-exception difference and the exception difference in the append block are extracted respectively, and the exception difference is inserted into the corresponding position according to the exception difference position bitmap to restore the complete ID difference sequence of the current encoding batch.

7. The method for multi-dimensional construction and retrieval of electronic meeting archives across all scenarios according to claim 1, characterized in that, The step of determining the intersection sparsity of the search term set based on the intersection result includes: calculating the number of common intersection elements of the absolute document ID sets corresponding to each search term; Calculate the number of elements in the union of the absolute document ID sets corresponding to each search term; The intersection-union ratio is obtained by dividing the number of elements in the common intersection by the number of elements in the union. The difference between the value 1 and the intersection-union ratio is determined as the intersection sparsity corresponding to the set of search terms.

8. The method for multi-dimensional construction and retrieval of electronic meeting archives across all scenarios according to claim 1 or 7, characterized in that, The global profile data includes historical intersection sparsity smoothing values; the determination of the intersection strategy based on the global profile data in the system includes: when the historical intersection sparsity smoothing value is less than a first threshold, determining to perform intersection using a sequential traversal method; when the historical intersection sparsity smoothing value is greater than or equal to the first threshold and less than a second threshold, determining to perform intersection using a jump search method; when the historical intersection sparsity smoothing value is greater than or equal to the second threshold, determining to perform intersection using a block-level jump traversal method. The process of updating the global profile data includes: performing a weighted calculation on the intersection sparsity and the current historical intersection sparsity smoothing value recorded in the global profile data to obtain the updated historical intersection sparsity smoothing value and storing it in the global profile data.

9. The method for multi-dimensional construction and retrieval of electronic meeting archives across all scenarios according to claim 1, characterized in that, The extraction of structured metadata based on meeting data includes: extracting text, timestamp records, and speaker identity information from the collected multidimensional audiovisual data stream; and mapping the extracted text, the timestamp records, and the speaker identity information as feature keys of different dimensions to the multidimensional inverted index.

10. The method for multi-dimensional construction and retrieval of electronic meeting archives across all scenarios according to claim 9, characterized in that, The extraction of structured metadata based on meeting data further includes: extracting specified business entities and meeting topics from the text; correspondingly, the construction of initial archive records includes: combining the meeting topics, the text, the timestamp records, the speaker's identity information, and the specified business entities to construct the initial archive records, and establishing a multi-dimensional search entry for the document ID list based on the specified business entities and meeting topics.

Images