Universal unstructured data index construction method and device and medium

Abstract

The invention discloses a universal unstructured data index construction method and device and a medium, and is suitable for multi-modal data such as pictures, videos and texts. The method comprises the steps that adaptive deep learning models are adopted for feature extraction for different modal data; building a multi-level cluster structure based on a recursive clustering method of automatic anchor point selection and hyperplane division; automatically generating a semantic abstract for each cluster by using a large language model; an index structure is efficiently organized through information such as categories, clusters and abstracts, and multi-dimensional retrieval is supported; and during query processing, a discrimination model is trained at low cost by using a very small amount of existing labels in the clustering clusters, so that efficient clustering cluster screening and multi-class combined query are realized, and a user does not need to specify positive and negative samples. The method has the advantages of being high in automation degree, low in labeling cost, high in expandability, capable of supporting multi-modal data, unique in labeling inheritance and the like, and the index construction and retrieval efficiency of large-scale unstructured data is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the fields of database management, artificial intelligence, information retrieval and large model applications, and in particular to a universal unstructured data index construction method, device and medium. Background Art

[0002] With the rapid development of the internet, the Internet of Things, and multimedia technologies, unstructured data (such as images, videos, text, and audio) is experiencing explosive growth across all industries. According to statistics, over 90% of newly generated data worldwide is unstructured. This data exists in a wide range of scenarios, including social media, surveillance systems, medical imaging, enterprise documents, and scientific research data. It is characterized by large volume, diverse types, complex structure, and rich semantics.

[0003] However, traditional structured data indexing and retrieval methods (such as B+ trees, inverted indexes, and hash indexes) are primarily targeted at tabular, relational databases and are difficult to directly apply to high-dimensional, heterogeneous, and semantically complex unstructured data. Existing unstructured data management solutions often suffer from the following deficiencies: Designed for a single modality or specific scenario, it lacks versatility and is difficult to adapt to the management and retrieval needs of multiple data types; The retrieval method is mainly based on keywords and tags, which makes it difficult to achieve efficient retrieval based on content and semantics, and cannot meet the needs of intelligent applications; The lack of an efficient index structure leads to inefficient large-scale data retrieval and high response latency, making it difficult to support real-time or near-real-time business scenarios. Existing solutions are mostly manual or semi-automatic processes in index construction, clustering, summarization, etc., with low levels of automation and intelligence, making it difficult to adapt to the continuous growth and dynamic changes of data.

[0004] With the development of deep learning and large-scale model technologies, neural network-based capabilities such as feature extraction, clustering, and semantic understanding have significantly improved, providing new ideas for the management and retrieval of unstructured data. However, how to systematically and automatically apply these capabilities to the indexing and retrieval of large-scale multimodal data still faces many challenges. Summary of the Invention

[0005] The purpose of this invention is to address the problems of low retrieval efficiency, poor index generalization, and high manual annotation costs of existing unstructured data (such as videos, pictures, text, etc.). A universal unstructured data index construction method, device, and medium are proposed to achieve efficient feature extraction, intelligent clustering, semantic summary generation, and automated query processing of multimodal data, thereby improving the management and retrieval capabilities of large-scale unstructured data.

[0006] The object of the present invention is achieved through the following technical solutions: A general method for constructing an unstructured data index includes the following steps: (1) Preprocess the acquired raw unstructured data and use an adapted deep learning model to extract features for different types of unstructured data to obtain high-dimensional feature vectors; (2) A recursive clustering method based on automatic anchor point selection and hyperplane partitioning is used to perform unsupervised grouping of high-dimensional feature vectors to form clusters with semantic consistency; (3) Using a large language model to automatically generate a semantic summary for each cluster and associate the summary with the cluster; (4) Generate an independent index unit for each cluster, and realize the automatic screening of original unstructured data through the retrieval methods of selection query, multi-category combination query, aggregation query, and restriction query.

[0007] Furthermore, in step (1), the unstructured data includes image, video and text data; for image data, a convolutional neural network is used to extract the visual features of the image and output a fixed-length high-dimensional feature vector to represent the content, structure and semantic information of the image; for video data, a temporal neural network or a multi-frame fusion model is used to extract the global temporal features of the video clip and output a feature vector that can reflect the dynamic content of the video; for text data, a large language model or an embedding model is used to extract the contextual semantic features of the text and output a feature vector that can reflect the text theme and semantic association; wherein all feature vectors are standardized.

[0008] Furthermore, step (2) includes the following sub-steps: (2.1) In each round of cluster splitting, a representative semantic anchor point is selected from the samples of the current node as a candidate anchor point; the optimal anchor point is determined by calculating the similarity distribution between the candidate anchor point and other samples, taking into account the semantic consistency and distinguishability; (2.2) Using the optimal anchor point as a reference, calculate the similarity of all samples to the optimal anchor point and divide the samples into high-similarity and low-similarity groups. After removing outliers from the high-similarity and low-similarity groups, calculate the mean vector within each group and normalize the difference between the two mean vectors to obtain the main direction. Furthermore, generate multiple auxiliary directions orthogonal to the main direction and form a candidate direction pool by screening through orthogonality and dispersion. (2.3) For each candidate direction, traverse multiple split points and project the samples into two groups, the left and the right. Use the separability, intra-group consistency, and semantic discrimination indicators to evaluate each "direction + split point" combination, and select the "direction + split point" with the highest comprehensive score as the optimal hyperplane for the current node. (2.4) With the entire dataset as the root node, recursively perform anchor point selection, direction generation, optimal hyperplane selection, and splitting operations. Each split divides the sample into two left and right child nodes, which are recursively processed separately until the termination condition is met. The recursive process forms a binary tree structure based on the hyperplane splitting, with each leaf node corresponding to a final cluster. (2.5) For all final clusters, classify them into different types based on the number of samples and internal consistency scores, identify abnormal nodes and optimize their clustering structure; (2.6) For each final cluster, count its semantic and mathematical features.

[0009] Furthermore, in sub-step (2.1), the selection of representative anchor points is based on the category distribution, density score, and semantic diversity of the samples, with priority given to samples containing high-frequency categories, high density, and strong diversity.

[0010] Furthermore, in sub-step (2.2), the cosine similarity between all samples in the current node and the selected anchor point is calculated, and the samples are sorted from high to low according to the similarity. Then, according to the configured sampling ratio parameter, a fixed proportion of samples are selected from the front and back ends of the sorting results to form high similarity groups and low similarity groups.

[0011] Furthermore, in sub-step (2.3), a quantile-based adaptive segmentation point selection strategy is adopted to project all samples in the candidate direction to obtain a one-dimensional projection value; multiple candidate segmentation points are uniformly sampled within the configured quantile range; for each candidate segmentation point, the samples are divided into two groups, the left and the right, based on whether the sample projection value is less than or equal to the segmentation point.

[0012] Furthermore, step (3) includes: organizing the statistical information of the cluster into a structured summary prompt, including the main categories and their proportions, a brief description of representative samples, the total number of samples and category distribution; inputting the structured summary prompt into the large language model to automatically generate a natural language summary; and then associating the generated summary text with the unique identifier of the corresponding cluster.

[0013] Furthermore, step (4) includes: generating an independent index unit for each cluster, including cluster ID, sample path, cluster statistical features, semantic summary, main category distribution and annotation information; constructing an inverted index from category to cluster, supporting the location of related clusters by category; providing retrieval methods such as selection query, multi-category combination query, aggregation query and restriction query, and realizing automatic screening of original unstructured data; The selection query initiates a query through categories and keywords, uses the inverted index and clustering structure to exclude irrelevant data, searches only within the relevant clusters, narrows the candidate set, and uses the existing annotation information within the clusters to train the SVM model to screen and sort the pictures or samples within the candidate range; the multi-category combination query initiates a query by combining multiple categories or keywords, and returns a set of pictures that meet multiple conditions at the same time; the aggregation query initiates a query through a specified category, uses the inverted index to count the number of samples and category distribution within the relevant clusters, and realizes aggregate statistical analysis; the restriction query initiates a query by limiting the number or range of returned results, and after the search results are sorted, outputs a specified number of samples according to the set threshold.

[0014] A general unstructured data index construction device for implementing the above method includes: The feature extraction module is used to preprocess the acquired raw unstructured data. For different types of unstructured data, it uses an adapted deep learning model to extract features and obtain high-dimensional feature vectors. The clustering grouping module is used to perform unsupervised grouping of high-dimensional feature vectors using a recursive clustering method based on automatic anchor point selection and hyperplane partitioning to form clusters with semantic consistency; Summary generation module, which uses a large language model to automatically generate a semantic summary for each cluster and associates the summary with the cluster; The index construction and query processing module is used to generate an independent index unit for each cluster, and realize the automatic screening of original unstructured data through the retrieval methods of selection query, multi-category combination query, aggregation query, and restriction query.

[0015] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned general unstructured data index construction method.

[0016] The present invention has the following beneficial effects: 1. The recursive clustering technology using automatic anchor point selection and hyperplane partitioning can better adapt to the data distribution in high-dimensional feature spaces compared to traditional clustering methods such as K-means, forming clusters with stronger semantic consistency, improving retrieval efficiency and accuracy, and solving the problem of slow retrieval speed of traditional indexing methods on large-scale unstructured data sets; 2. The technology of automatically generating cluster summaries using a large language model can provide richer semantic information than traditional indexing methods that rely solely on keywords or tags, enhancing the interpretability of search results and solving the problem of difficult to understand and lack of semantic relevance in existing search results. 3. The technology of combining a small amount of annotation within clusters with online SVM training significantly reduces the cost of manual annotation compared to traditional methods that require a large number of labeled samples, while maintaining good retrieval results and solving the practical problem of high annotation costs for large-scale datasets. 4. A unified multimodal processing framework, compared to indexing methods designed for a single modality, can process different types of data, such as images, videos, and text, in the same system, reducing system complexity and maintenance costs and solving the problem of difficult multimodal data management. 5. The modular system design makes the interfaces between components clear, facilitates subsequent expansion and integration of new functions, enhances the maintainability and scalability of the system, and adapts to the continuous growth and dynamic changes of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive efforts. Figure 1 A schematic diagram of the hyperplane clustering process provided by an embodiment of the present invention; Figure 2 A schematic diagram of the association between summary generation and clustering provided by an embodiment of the present invention; Figure 3 Schematic diagram of the index structure and inverted index provided by an embodiment of the present invention; Figure 4 A schematic diagram of the query processing flow provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The present invention will be described in detail below with reference to the accompanying drawings. Unless there is any conflict, the features of the following embodiments and implementations may be combined with each other.

[0019] The present invention provides a general unstructured data index construction method suitable for scenarios such as intelligent retrieval, content management, and data analysis, and specifically includes the following steps: (1) Obtaining raw unstructured data, preprocessing the raw unstructured data, and extracting features; using an adapted deep learning model for feature extraction for different types of unstructured data, the raw unstructured data is converted into high-dimensional feature vectors. Unlike existing technologies, this invention not only focuses on the surface features or simple mathematical representations of the data, but also digs deeper into the semantic connotations of the data, achieving true semantic understanding and retrieval. This semantic-based processing method is one of the core innovations of this invention that distinguishes it from existing technologies, enabling the system to understand the intrinsic meaning of the data rather than relying solely on surface feature matching.

[0020] The present invention can acquire raw unstructured data from a variety of sources: image data can come from enterprise product libraries, product images from e-commerce platforms, photos shared on social media, and visual data collected by surveillance systems; video data can come from online video platforms, educational and training resource libraries, corporate promotional materials, security monitoring systems, and the like; and text data can come from enterprise knowledge bases, news media reports, social media comments, customer feedback, technical documentation, and the like. For example, in a retail industry application scenario, the system can process product images from a product gallery, promotional videos from the marketing department, and user comment text from the customer service system, constructing a common index structure for this multimodal data and supporting multimodal intelligent retrieval services.

[0021] (1.1) Preprocessing of raw unstructured data: Specifically, this step addresses the diversity and complexity of the raw unstructured data by performing unified preprocessing operations to ensure the effectiveness and consistency of subsequent feature extraction and clustering. Unlike existing methods designed for a single modality or specific scenario, this invention provides a universal preprocessing framework that can simultaneously process a variety of unstructured data, including images, videos, and text, addressing the lack of versatility of existing solutions.

[0022] Furthermore, image data is subjected to format unification (such as JPEG, PNG, etc.), size normalization, color space conversion, and denoising processing to eliminate differences in resolution, color, and noise between images from different sources. Video data is subjected to frame extraction (such as equal-interval sampling or keyframe extraction), resolution unification, and frame rate standardization to ensure the temporal consistency and representativeness of video features. Text data is subjected to word segmentation, stop word removal, unified encoding format, and removal of redundant and invalid characters to improve the processability and semantic purity of text content.

[0023] Among them, the above preprocessing operations can be flexibly configured according to the data type and actual needs to ensure that all data samples have a unified input standard before entering the feature extraction stage.

[0024] For example, in an enterprise media resource library containing multimodal data, the system needs to process heterogeneous data from different departments, devices, and time periods. For image data, the system converts various formats (BMP, TIFF, RAW, etc.) into JPEG format and standardizes the resolution to 1024×768 pixels to ensure consistency in subsequent feature extraction. For video data, the system extracts one keyframe per second from each video and standardizes the resolution of all videos to 720p and 30fps for subsequent processing. For text data, the system converts various formats (DOC, PDF, TXT, etc.) into plain text, performs Chinese and English word segmentation, removes stop words and special characters, and preserves the core semantic content. Through this unified preprocessing framework, the system can convert raw data of varying formats and quality into standardized input, laying the foundation for subsequent feature extraction and index construction, significantly improving the robustness and versatility of the retrieval system.

[0025] (1.2) Feature extraction: Specifically, this step uses an adapted deep learning model (such as a convolutional neural network, a temporal neural network, a large language model, an embedding model, etc.) to extract features for different types of unstructured data, converting the original unstructured data samples into high-dimensional feature vectors, i.e., multimodal feature vectors. This solves the problem that the existing retrieval methods based on keywords and tags are difficult to achieve efficient retrieval based on content and semantics. The high-dimensional semantic feature vectors extracted by the deep learning model of the present invention can more comprehensively capture the semantic information of the data content, laying the foundation for subsequent intelligent retrieval. Unlike the existing technology that mainly relies on manually defined features or shallow features, the present invention uses high-dimensional features automatically learned and extracted by deep learning models to more accurately express the semantic connotation of the data, achieving a fundamental shift from "surface matching" to "semantic understanding."

[0026] Furthermore, for image data, convolutional neural networks (such as ResNet, etc.) are used to extract the visual features of the image and output fixed-length high-dimensional feature vectors that can represent the content, structure and semantic information of the image; for video data, temporal neural networks (such as 3D convolutional networks, temporal Transformer, etc.) or multi-frame fusion models are used to extract the global temporal features of video clips and output feature vectors that can reflect the dynamic content of the video; for text data, large language models (such as BERT, etc.) or embedding models are used to extract the contextual semantic features of the text and output feature vectors that can reflect the text theme and semantic association.

[0027] Among them, all feature vectors are standardized (such as mean variance normalization and L2 normalization) to ensure the accuracy of subsequent clustering and retrieval.

[0028] For example, in an intelligent image retrieval system, when a user uploads a query image containing a "beach sunset" scene, traditional keyword-based retrieval methods may only match images manually labeled as "beach" or "sunset," failing to identify visually similar but incorrectly labeled images. The present invention uses a 2048-dimensional feature vector extracted by a ResNet-50 deep convolutional network to capture rich visual elements in the image, such as the orange-red sky, reflections on the water, and the texture of the beach. Similarly, for video data, the system uses a 3D-CNN model to extract spatiotemporal features from a video of a "street basketball game," not only identifying the characters and basketball in the frame but also capturing action sequences such as shooting and dribbling. For text data, the system uses the BERT model to convert an article on "AI ethics" into a 768-dimensional semantic vector. This vector encodes not only the article's topic but also its argument, sentiment, and context. This deep learning-based feature extraction method enables the system to understand the intrinsic semantics of the data, enabling truly content-based intelligent retrieval, significantly surpassing traditional keyword-based retrieval methods.

[0029] (2) Hyperplane clustering: A recursive clustering method based on automatic anchor point selection and hyperplane partitioning is used to perform efficient unsupervised grouping of all high-dimensional feature vectors to form clusters with good semantic consistency. This innovative design solves the problem of low efficiency and high response delay in large-scale data retrieval caused by the lack of efficient index structure in the existing technology. By constructing a multi-level semantic clustering structure, the retrieval complexity is significantly reduced, providing support for real-time or near-real-time business scenarios. Unlike the distance-based clustering methods such as K-means used in the existing technology, the hyperplane clustering method of the present invention is more suitable for the characteristics of high-dimensional semantic space, can better capture the semantic structure of data, and form more interpretable clustering results. See Figure 1 The hyperplane clustering process diagram of the present invention shows in detail the complete processing flow from the original feature vector to the final clustering result. The process includes the following eight key steps: The first step is to generate "raw feature vectors (image / video / text features)"—high-dimensional feature vectors extracted from data of different modalities—as input to the clustering algorithm. These feature vectors contain semantic information about the image, video, or text data and serve as the basis for subsequent clustering analysis.

[0030] The second step, "preprocessing and normalization," standardizes the raw feature vectors to eliminate scale differences across feature dimensions and improve the accuracy and stability of subsequent clustering. This step ensures that feature vectors from different modalities are comparable.

[0031] The third step, "Semantic Anchor Point Selection (Representative Sample Sampling, Category / Density / Diversity Analysis)," selects the most representative samples from the current dataset as anchor points. This step analyzes the sample's category distribution, density score, and diversity characteristics to select the sample points that best represent the data distribution characteristics, providing a reference benchmark for subsequent hyperplane partitioning.

[0032] The fourth step, "Hyperplane Direction Generation (Main Direction + Orthogonal Auxiliary Directions)," calculates the main direction that best distinguishes the data based on the selected anchor points and generates auxiliary directions orthogonal to the main direction. This combination of main and auxiliary directions enhances the hyperplane's segmentation capability and adaptability.

[0033] The fifth step, "Optimal Hyperplane Selection (Multi-Indicator Scoring, Joint Optimization of Split Points)", evaluates the generated candidate hyperplanes on multiple dimensions, including separability, intra-group consistency, and semantic discrimination. The optimal split point is then determined through joint optimization to achieve optimal data partitioning.

[0034] The sixth step, "Recursively Split / Construct Hyperplane Tree (Binary Tree Structure, Configurable Termination Condition)," recursively splits the dataset into two subsets, left and right, based on the selected hyperplane, forming a binary tree structure. This process continues until a pre-defined termination condition is met, such as when the number of node samples falls below a threshold, the tree depth reaches an upper limit, or the internal consistency of the nodes is sufficiently high.

[0035] The seventh step, "post-processing optimization (abnormal node correction, sample reallocation, and label attribution check)," optimizes the clustering results. This includes correcting abnormal nodes, reallocating boundary samples, and checking the attribution consistency of label information to improve the quality and practicality of the clustering results.

[0036] The eighth step, "output clustering results", outputs the final clusters and related information as the basis for subsequent index construction and summary generation.

[0037] Through the organic combination of these eight steps, the present invention achieves efficient, interpretable, and semantically consistent clustering grouping, laying a solid foundation for index construction of unstructured data.

[0038] For example, in a million-level image retrieval scenario, traditional linear scanning methods require calculating the similarity between the query image and all images in the database one by one, and the retrieval time increases linearly with the amount of data. However, the hyperplane clustering method of the present invention recursively divides the feature space into multiple semantically related subspaces, so that only a few relevant clusters need to be accessed during retrieval, reducing the retrieval complexity from O(n) to O(log n), and achieving a response time of seconds on a million-level dataset. The specific process is as follows: (2.1) Semantic anchor selection: Specifically, during each round of cluster splitting, representative "semantic anchors" are first selected from the samples of the current node as candidate anchors. The selection of representative anchors is based on the sample's category distribution, density score, and semantic diversity, with priority given to samples containing high-frequency categories, high density, and strong diversity.

[0039] By calculating the similarity distribution between the candidate anchor point and other samples, the optimal anchor point is finally determined by comprehensively considering the semantic consistency and discrimination ability.

[0040] This step ensures that each split is centered on the most representative sample, improving the semantic interpretability of the clustering.

[0041] For example, in an image retrieval system, when clustering a dataset containing images of multiple categories, the semantic anchor selection algorithm of the present invention prioritizes images that are centered within the category and have rich feature expression as anchors. For example, in a cluster node containing the "car" category, the system selects car images with typical viewing angles, high definition, and minimal background interference as anchors, rather than blurry, partially occluded, or angled images. This ensures that the subsequent hyperplane partitioning maximizes the preservation of the category's semantic integrity.

[0042] (2.2) Hyperplane direction generation: Specifically, using the optimal anchor point as a reference, the similarity of all samples to the optimal anchor point is calculated, and the samples are divided into high-similarity and low-similarity groups. First, the cosine similarity of all samples in the current node with the selected anchor point is calculated; then, the samples are sorted from high to low similarity; then, based on the configurable sampling ratio parameters (high_sim_ratio and low_sim_ratio), a fixed ratio (for example, 15%) of samples is selected from the front and back ends of the sorted results, respectively, to form the initial high-similarity and low-similarity groups. This sorting-based partitioning method is more adaptable than traditional fixed threshold partitioning and can adapt to different data distribution characteristics.

[0043] Furthermore, after removing outliers from the high and low similarity groups, the mean vector within the group is calculated, and the main direction is obtained by normalizing the difference vector of the two groups' means.

[0044] In order to improve the diversity and robustness of splitting, multiple auxiliary directions orthogonal to the main direction are further generated, and a candidate direction pool is formed through orthogonality and dispersion screening.

[0045] This step ensures that the hyperplane direction of each split reflects both the primary semantic distinction and diversity, avoiding local optima. This sophisticated outlier filtering strategy significantly improves the stability and semantic consistency of subsequent direction generation and is a key technical innovation in the present invention's hyperplane clustering direction generation process.

[0046] For example, in a text retrieval scenario, when processing a document collection encompassing multiple topics, the present invention's hyperplane direction generation algorithm first divides the documents into a high-similarity group (other sports news) and a low-similarity group (such as political or technology news) based on their similarity to an anchor document (e.g., a typical sports news article). It then calculates the semantic centers of the two groups and determines the primary direction that best distinguishes them. Simultaneously, the system also generates multiple auxiliary directions, such as those that distinguish between different sports (e.g., football vs. basketball). This multi-directional strategy enables the final clustering structure to capture multiple dimensions of text semantics, significantly improving retrieval accuracy.

[0047] (2.3) Optimal hyperplane selection: Specifically, for each candidate direction, multiple split points are traversed, and the samples are projected and divided into two groups, the left and the right. This method employs a quantile-based adaptive split point selection strategy: First, all samples are projected onto the candidate direction to obtain a one-dimensional projection value. Then, multiple candidate split points (typically 20-50 points) are uniformly sampled within a configured quantile range (e.g., from the 10th to the 90th percentile). For each candidate split point, the samples are divided into two groups, the left and the right, based on whether the sample projection value is less than or equal to the split point. This quantile-based partitioning method can adapt to different data distribution characteristics, avoid extreme splitting situations (such as overly unbalanced partitioning), and ensure that both the left and the right groups have sufficient samples for subsequent evaluation.

[0048] Furthermore, the present invention designs a semantic-driven multi-dimensional scoring mechanism, which uses the comprehensive scoring formula Score(w)=γ·(μ L -μ R )² / (σ L ²+σ R ²)+β·η+(1-γ-β)·δ, and use multiple indicators such as separability, intra-group consistency, and semantic distinction to evaluate each set of "direction+split point" combinations. Among them, the separability term (μ L -μ R )² / (σ L ²+σ R ²) measures the projection distinction between the left and right subclusters in direction w, μ L and μ R are the means of the left and right groups in the projection direction, σ L ² and σ R² represents the variance of the left and right groups, respectively. The internal consistency term η is evaluated by calculating the average cosine similarity between samples within the left and right child nodes and incorporating a sample size balance factor. The semantic discrimination term δ evaluates the hyperplane's ability to distinguish different semantics by measuring the difference in similarity between the left and right groups and the anchor point, as well as the semantic difference between the left and right groups. γ, β, and (1-γ-β) are configurable weight parameters used to balance the importance of the three scoring dimensions.

[0049] After multiple rounds of evaluation and sampling, the "direction + split point" with the highest comprehensive score is finally selected as the optimal hyperplane for the current node.

[0050] This step ensures that each split maximizes between-group variance, within-group consistency, and semantic interpretability.

[0051] The innovations of this invention in the hyperplane selection process include: (1) adopting an adaptive segmentation point scanning strategy to dynamically determine candidate segmentation points through the configured quantile range, avoiding the suboptimal partitioning that may be caused by fixed segmentation points; (2) introducing an efficient evaluation mechanism based on subsampling, first performing random subsampling and then evaluating large-scale data sets, significantly improving computational efficiency; (3) designing a semantically enhanced intra-group consistency evaluation method, which not only considers the average similarity but also introduces the tightness of the similarity distribution as a supplementary indicator of clustering quality; (4) developing a multi-level semantic discrimination evaluation framework, which ensures that the segmentation results have clear semantic interpretation through a weighted combination of inter-group semantic differences and anchor point alignment differences. In the actual implementation process, the system first performs a preliminary evaluation of all candidate directions and selects the directions with the highest scores; then, these top candidate directions are accurately evaluated using the full data, and finally the "direction + segmentation point" combination with the highest comprehensive score is selected as the optimal hyperplane.

[0052] For example, in a video retrieval system, when a video collection containing different content types needs to be partitioned, the present invention evaluates multiple possible partitioning schemes. For example, the system might attempt to partition according to different semantic dimensions, such as "indoor vs. outdoor," "people vs. scenery," or "static vs. dynamic," and calculate a score for each partitioning scheme. Assuming the "people vs. scenery" partitioning scheme receives the highest score, the system will select this hyperplane as the basis for splitting the current node. This ensures that the generated clusters have clear semantic boundaries, making it easier for users to quickly locate target videos based on semantic descriptions.

[0053] (2.4) Recursively construct the hyperplane tree: Specifically, taking the entire dataset as the root node, anchor point selection, direction generation, optimal hyperplane selection and splitting operations are recursively performed.

[0054] Furthermore, each split divides the sample into two left and right child nodes, which are recursively processed separately until the termination condition is met (such as the number of node samples is less than the threshold, the tree depth reaches the upper limit, the internal consistency of the node is high, etc.).

[0055] The recursive process forms a binary tree structure based on the hyperplane splitting basis, and each leaf node corresponds to a final cluster.

[0056] This structure can adapt to data distribution and carefully depict the semantic structure in high-dimensional space.

[0057] For example, in a large-scale image retrieval system, the present invention begins with a root node containing millions of images and constructs a multi-level hyperplane tree through recursive splitting. For example, the first level of splitting might classify images into two categories: "natural scenes" and "artificial scenes." The second level of splitting might further subdivide "natural scenes" into "mountains and waters" and "plants and animals," and "artificial scenes" into "architecture" and "vehicles." This process continues until leaf nodes with sufficiently high semantic consistency are formed. When a user queries for images like "mountain streams," the system simply follows the tree structure to quickly locate the relevant leaf node, without having to scan the entire dataset. This significantly improves retrieval efficiency.

[0058] (2.5) Post-processing optimization: Specifically, all leaf nodes (final clusters) are classified based on sample size and internal consistency scores to identify outliers. This paper designs a comprehensive node classification and outlier handling mechanism. First, a node classifier is used to classify all leaf nodes in two dimensions: small, medium, and large nodes based on sample size, and anomaly, good, and excellent nodes based on consistency. The classification criteria are based on configurable threshold parameters, such as small_threshold_factor, medium_threshold_factor, and coherence_threshold, ensuring that the classification process can adapt to the characteristics of different datasets.

[0059] Furthermore, the clustering structure of abnormal nodes is optimized by using methods such as sample redistribution and node merging to ensure high internal consistency of each cluster and reasonable attribution of abnormal samples. The abnormal node processor of the present invention adopts an iterative processing strategy, giving priority to processing small abnormal nodes, then medium-sized abnormal nodes, and finally large abnormal nodes. For small abnormal nodes, the system will transfer all its samples to the nodes with the most similar semantics; for medium-sized abnormal nodes, the system will transfer a certain proportion (such as 20%) of samples; for large abnormal nodes, the system will transfer a smaller proportion (such as 10%) of samples. During the sample transfer process, the system will give priority to transferring those samples (outlier samples) with low semantic consistency with the node as a whole, and find the best attribution node for each sample by calculating the similarity between the sample and the target node. In particular, the present invention implements a synchronous migration mechanism of annotation information during the sample transfer process to ensure that the annotation attribution of the sample always remains consistent, avoiding the loss or conflict of annotation information.

[0060] This step uses multiple rounds of iterative optimization until no abnormal nodes are found or the maximum number of iterations is reached, significantly improving the stability, consistency, and practical application of clustering results. The system calculates node consistency statistics before and after processing to quantitatively evaluate the optimization effect, providing a high-quality clustering foundation for subsequent index construction and query processing.

[0061] For example, in a text retrieval system, after clustering, some abnormal nodes may appear, such as "long-tail nodes" with too few samples or "hybrid nodes" with low internal consistency. The present invention optimizes these nodes, for example by merging small clusters containing fewer than 10 documents into larger, semantically similar clusters, or further splitting hybrid clusters with widely varying document topics. Through this post-processing optimization, the system can generate a more balanced and semantically pure cluster structure, improving the relevance of search results and user experience.

[0062] (2.6) Cluster feature statistics and annotation inheritance: Specifically, for each final cluster, its semantic and mathematical features are statistically analyzed to facilitate subsequent summary generation and analysis.

[0063] Furthermore, semantic features mainly rely on a very small amount of manual annotation information. By extracting a very small number of samples from each cluster for annotation, the main category distribution and representative semantic description of the cluster can be obtained, which greatly reduces the cost of manual annotation. The present invention adopts an adaptive sampling strategy, which only extracts 10% of the samples in each cluster for annotation by default (at least 10 samples). This ratio is sufficient to obtain accurate category distribution statistics in most scenarios. For example, for a cluster containing 500 samples, the system only needs to annotate 50 samples; and for small clusters (such as only 50 samples), the system will also ensure that at least 10 samples are annotated to ensure statistical reliability. In addition, the present invention designs an annotation inheritance mechanism, which gives priority to the use of existing annotation information and only supplements new annotations when necessary, further improving the annotation efficiency.

[0064] Mathematical features reflect the distribution structure and diversity within the cluster by performing statistical analysis on the feature vectors of all samples in the cluster, including mean, standard deviation, principal component analysis (PCA), etc.

[0065] The clustering process supports the inheritance and migration of annotation information to ensure that the annotation of each sample is unique and complete. After operations such as sample redistribution and node merging, the annotation information is migrated synchronously with the sample to ensure the integrity and consistency of the annotation data.

[0066] For example, in an image retrieval system, for a "cat" cluster containing 500 images, the present invention only needs to randomly select 25 images (5%) for manual annotation to obtain the main category distribution of the cluster (e.g., "domestic cat" accounts for 60%, "wild cat" accounts for 30%, and "other cats" accounts for 10%). Simultaneously, the system calculates mathematical features such as the mean and covariance of the feature vectors of these 500 images for subsequent similarity calculation and query processing. If certain images in the cluster are reassigned to other clusters due to post-processing optimization, their annotation information is transferred accordingly, ensuring that the annotation status of each image in the system remains consistent, avoiding duplicate annotations and annotation conflicts.

[0067] This step achieves efficient, interpretable, and semantically consistent clustering in high-dimensional feature space through mechanisms such as recursive hyperplane partitioning, anchor-driven direction generation and splitting, and post-processing optimization, laying a solid foundation for subsequent summary generation and index construction.

[0068] (3) Large model summary generation: see Figure 2, using generative artificial intelligence such as large language models, automatically generates a semantic summary for each cluster, and associates the summary with the cluster. This is one of the core innovations of the present invention, which organically combines large model technology with traditional index construction methods to achieve a highly intelligent and interpretable index structure. Compared with the existing indexing methods that rely only on keywords or tags, the present invention innovatively uses the super data analysis capabilities of large models to generate comprehensive semantic summaries for data, accurately describes the data content, and greatly improves the comprehensibility of retrieval results and user experience. The schematic diagram of the association between summary generation and clusters of the present invention shows in detail the complete process from multiple clusters to generating semantic summaries and establishing associations. The process includes the following six key steps: In the first step, the dashed box at the top of the figure shows "multiple clusters," including "Cluster 1 (sample paths, feature vectors, sampling annotations)," "Cluster 2," and finally "Cluster N." Each cluster represents a set of semantically similar data samples formed through hyperplane clustering. It records the sample paths, feature vectors, and a small amount of sampling annotation information.

[0069] In the second step, the information of these clusters is collected in the "Statistical Analysis (Vector Mathematical Features, Sampling and Annotation Features)" module, where the mathematical and annotation features of each cluster are statistically analyzed to provide an objective data basis for subsequent summary generation.

[0070] In the third step, based on the statistical analysis results, the system generates a "structured summary prompt", which organizes the key information of the cluster into a structured prompt template to facilitate understanding and processing by the large language model.

[0071] In the fourth step, the structured prompts are input into the "big language model", which uses its powerful language understanding and generation capabilities to semantically parse and summarize the characteristics of the clusters.

[0072] In the fifth step, the large language model generates multiple semantic summary texts based on the input structured prompts, including "Semantic Summary Text 1", "Semantic Summary Text 2" and even "Semantic Summary Text N". Each summary text describes the semantic features of the cluster from a different perspective.

[0073] In the sixth step, the system establishes an "association between summary and cluster ID" and establishes a unique mapping relationship between the generated semantic summary text and the corresponding cluster, ensuring that each cluster has a corresponding semantic description, which facilitates subsequent index construction and query processing.

[0074] Through the organic combination of these six steps, the present invention achieves high-quality semantic interpretation and expression of clustering results, greatly improving the intelligence and usability of unstructured data indexing.

[0075] Specifically, this step uses a large language model to automatically generate summary text for each cluster that can reflect the main semantic features of the cluster, thereby improving the interpretability of the index and the semantic relevance of downstream retrieval. This innovative design solves the problem that the existing technology is mostly manual or semi-automatic processes in index construction, clustering, summary and other links, and has low automation and intelligence levels. By introducing a large language model, highly automated summary generation is achieved, and high-quality semantic descriptions can be obtained without human intervention, greatly improving the system's ability to process large-scale data. Compared with the existing technology, the large model summary generation of the present invention can not only significantly reduce labor costs, but also provide richer and more accurate semantic descriptions, so that non-professional users can easily understand and use the search results. The specific process is as follows: (3.1) Cluster statistical analysis: Specifically, this step directly uses key information such as the main category distribution, representative samples, and sample size obtained in the previous step of cluster feature statistics without repeated calculations.

[0076] For example, in an image retrieval system, for a cluster containing landscape images, the system extracts the following statistical information: category distribution (e.g., "mountains" accounts for 45%, "beach" accounts for 30%, and "forest" accounts for 25%), representative samples (e.g., the five images closest to the cluster center), and the total number of samples (e.g., the cluster contains 200 images). These objective statistics provide a factual basis for subsequent summary generation, ensuring that the generated summary accurately reflects the content composition of the cluster, thereby supporting more precise retrieval.

[0077] These statistical information provides a comprehensive and objective data basis for summary generation, ensuring efficient process connection and data consistency.

[0078] (3.2) Summary Tips Construction: Specifically, the statistical information of the clusters is organized into structured summary prompts, including: main categories and their proportions, brief descriptions of representative samples, total number of samples, category distribution, etc.

[0079] Furthermore, the summary prompt can be organized in a templated manner, such as "This cluster mainly includes category A (accounting for X%), category B (accounting for Y%), the representative samples are..., and the total number of samples is N."

[0080] Through structured prompts, we ensure that the large language model can accurately understand the core semantics and statistical features of the clusters.

[0081] For example, in a text retrieval system, for a cluster containing news articles, the system will organize the statistical information into the following structured prompt: "This cluster contains 120 news articles, the main categories are: sports news (accounting for 65%), event reports (accounting for 25%), and athlete interviews (accounting for 10%). Representative article titles include '2023 World Cup Final Analysis', 'Top Player Season Performance Evaluation', etc. The main keywords include: football, game, champion, player, tactics"; this structured prompt enables the large language model to accurately grasp the core content and semantic features of the cluster, thereby generating a more accurate summary.

[0082] (3.3) Large Language Model Summary Generation: Specifically, the structured summary prompt is input into a large language model to automatically generate a natural language summary.

[0083] Furthermore, the summary generation process can adopt a batch processing method to improve the processing efficiency under large-scale data sets.

[0084] The generated summary should cover key information such as the main categories of the cluster, representative samples, sample size, category distribution, etc. The language expression should be concise, accurate and readable.

[0085] If the summary generation result does not meet expectations, quality control can be performed by adjusting the prompt content or adding constraints.

[0086] For example, in a video retrieval system, after the system inputs the structured prompt of a cluster into a large language model, it may generate the following summary: "This is a collection of 48 educational videos, mainly scientific experiment demonstrations (accounting for 70%) and explanations of natural phenomena (accounting for 30%). The videos are mostly based on middle school physics and chemistry experiments, including detailed experimental steps, principle explanations and result analysis. Most videos are between 5 and 10 minutes long and are suitable as teaching auxiliary materials." This natural language summary is richer and more specific than simple category labels, which can help users quickly understand the content characteristics of the cluster and improve the accuracy and efficiency of retrieval.

[0087] (3.4) Associative storage of summaries and clusters: Specifically, the generated summary text is associated with the unique identifier of the corresponding cluster to ensure that each cluster has an independent and traceable semantic description.

[0088] The summary information, clustering results, and statistical information are included in the index structure to facilitate subsequent multi-dimensional retrieval and analysis.

[0089] For example, in an image retrieval system, each cluster is assigned a unique identifier (e.g., "cluster_id_0042") and a persistent association is established between the generated summary (e.g., "This is a set of night photos taken of city streets, primarily featuring neon lights, traffic patterns, and urban architecture. Most photos utilize long exposure techniques, creating a visual effect of flowing light and shadow"). This association is stored in the system's index structure (e.g., a JSON-formatted index file). When a user searches for keywords like "city nightscape" or "neon lights," the system can quickly locate the relevant clusters and further verify their relevance using the summary information, improving retrieval accuracy.

[0090] This step achieves high-quality semantic interpretation and expression of clustering results through meticulous statistical analysis, structured prompt construction, automatic summary generation, and a strict summary-clustering association mechanism, greatly improving the intelligence and practicality of unstructured data indexing.

[0091] (4) Index structure construction and query processing: Specifically, this step builds an efficient and scalable index structure around the clustering results and summary information, and designs an intelligent query processing process to give full play to the engineering advantages of the system. The present invention comprehensively solves many defects in the existing technology through this design: it solves the problem of lack of universality by supporting multimodal data processing; it solves the problem of difficulty in achieving efficient semantic retrieval through an index structure based on content and semantics; it solves the problem of low efficiency in large-scale data retrieval through an efficient hierarchical index structure; it solves the problem of high manual intervention costs through an automated query processing process, and realizes a truly universal, efficient and intelligent unstructured data indexing method. Compared with the existing technology, the index structure of the present invention not only focuses on the physical organization of the data, but also pays more attention to the expression and utilization of semantic associations. Through the organic combination of clustering clusters, summaries and inverted indexes, it realizes semantic-driven efficient retrieval, which is another core innovation that distinguishes the present invention from the existing technology.

[0092] (4.1) Index structure construction: see Figure 3 The index structure and inverted index diagram of the present invention details the organization of index units and their relationship with query processing. The diagram includes the following five key parts: The first section, shown in the dashed box at the top of the figure, contains the "Cluster Summary Index Unit," which includes "Cluster Summary 1 (ID, Sample, Summary)," "Cluster Summary 2," and finally "Cluster Summary N." Each index unit contains a unique cluster identifier, a list of sample paths, and their corresponding semantic summary. These form the basic building blocks of the index structure.

[0093] In the second part, three paths extend from the index unit, connecting to the "Category A Inverted Index (Category A → Cluster, Summary 1, 2)", the "Category B Inverted Index (Category B → Cluster, Summary 1, N)", and the "Category C Inverted Index (Category C → Cluster, Summary 2, N)". This demonstrates that the system constructs an inverted mapping from categories to clusters. A category can be associated with multiple clusters, and a cluster can also contain multiple categories.

[0094] In the third part, these inverted indexes are aggregated into the "Query Request (such as retrieval by category)" node, which means that users can initiate queries by specifying categories, and the system will use the inverted index to quickly locate related clusters.

[0095] In the fourth part, the query request is processed by "quickly filtering according to the summary and locating relevant clusters". The system uses the semantic summary to preliminarily screen the candidate clusters and quickly narrow the search scope.

[0096] In the fifth part, the system finally "returns cluster samples / summary" as the query result, showing the user the data samples that meet the query conditions and their semantic descriptions.

[0097] The system generates an independent index unit (e.g., a JSON file) for each cluster, containing the cluster ID, sample path, cluster statistics, semantic summary, main category distribution, and annotation information. This structured index organization enables efficient data retrieval and management, enabling rapid response to user queries while maintaining good scalability.

[0098] Furthermore, an inverted index from category to cluster is constructed to support the rapid location of related clusters by category, thus achieving efficient category retrieval and statistics.

[0099] For example, in an image retrieval system, for a dataset containing photos of natural scenery, the system's index structure organizes semantically similar images into the same cluster. For example, a cluster of mountain scenery would have a unique identifier associated with the path information for all images in the cluster. The system would generate a descriptive summary for this cluster, such as "This is a collection of mountain scenery photos taken at high altitudes. The primary features are snow-capped peaks, blue skies, and expansive views. Most photos use wide-angle compositions, with predominantly blue and white tones." The system would also record the category distribution of the images in the cluster (e.g., 65% of the images contain mountain elements, 25% contain lake elements, and 10% contain forest elements), as well as information about a small number of annotated examples and the mathematical characteristics of the cluster. This organization allows the system to quickly locate the relevant cluster for a user query of "snow-capped mountain scenery" without having to retrieve all images individually.

[0100] All index data and clustering results are updated synchronously to ensure data consistency and real-time performance.

[0101] The index structure is highly modular, which facilitates subsequent expansion to support more retrieval dimensions or integrate new functions.

[0102] (4.2) Query processing flow: See Figure 4 The query processing flow diagram of the present invention shows in detail the complete processing process from user input to returning query results. The diagram includes three main query paths and multiple processing steps: First, at the top of the diagram is the "User Query Input" node, which represents the starting point for a user's query. Based on the user's input type, the system divides the query request into three different processing paths, each optimized for a specific query requirement: The first path is the "aggregate" process: 1. After a user enters an aggregate query, the system first performs an inverted index search to quickly locate clusters containing the target category. This step leverages the pre-built category-to-cluster mapping to avoid a full scan and significantly improve search efficiency.

[0103] 2. The system then performs "summary content statistics (such as category distribution and quantity)" on the retrieved results. The system will analyze the summary information of the clusters, extract statistical features such as category distribution and number of objects, and perform summary calculations.

[0104] 3. Finally, the "aggregate statistical results" are returned to the user, including statistical information such as the total number of categories, the total number of objects, and the category distribution ratio. This provides users with the overall distribution of the dataset and supports data analysis and decision-making.

[0105] The second path is the "select query" process: 1. After the user enters a query, the system performs an initial screening using the inverted index and summary information. This step quickly narrows the search scope and improves subsequent processing efficiency.

[0106] 2. After the initial screening, a "cluster candidate set" is obtained, which contains clusters that may meet the query criteria. The system extracts sample paths and feature vectors from these clusters to prepare for subsequent precise screening.

[0107] 3. The system then uses the "Intra-cluster SVM Discriminant Model (Automatically Trained with Annotations)" to automatically train a support vector machine model based on a small number of annotated samples within the cluster. This step is one of the core innovations of this invention. It can build an efficient discriminant model with a very small amount of annotated data, significantly reducing annotation costs.

[0108] 4. Then perform "high-precision screening and sorting", use the trained SVM model to accurately screen the candidate data, and sort them according to the relevance score to ensure that the most relevant results are ranked first.

[0109] 5. Finally, "return accurate search results" to users, providing high-quality result sets that meet the query intent, greatly improving the user experience.

[0110] The third path is the "limit query" process: 1. After the user enters a restricted query request, the system performs "initial screening of inverted index categories" to quickly locate clusters of related categories and establish a preliminary candidate set.

[0111] 2. Then apply "limit condition (confidence) filtering" to filter the candidate set according to the confidence threshold set by the user, eliminating samples with insufficient confidence to ensure the quality of the results.

[0112] 3. Then obtain the "most relevant candidate set", which contains the samples that best meet the conditions, in preparation for subsequent quantity restriction.

[0113] 4. Then perform the "Select + Truncate / Limit Quantity" operation to truncate the results according to the user-specified quantity limit to ensure that the number of returned results meets user requirements.

[0114] 5. Finally, "return a limited number of results" to the user, providing high-quality query results that meet the quantity limit.

[0115] These three query paths share the underlying feature vector management, cluster loading, and index structure, ensuring system consistency and efficiency. Through this design, the present invention can flexibly respond to different types of query requirements, provide efficient and accurate search services, and significantly improve user experience and system practicality. The system provides users with a variety of efficient search methods such as "select query", "multi-category combination query", "aggregate query", and "restriction query". The core advantages are reflected in: Select query (select): Specifically, users can initiate queries based on categories, keywords, and other criteria without having to understand the underlying index structure.

[0116] Furthermore, the system first uses the inverted index and clustering structure to quickly eliminate a large amount of irrelevant data, and only performs subsequent searches within the scope of relevant clusters, greatly narrowing the candidate set.

[0117] Within the candidate range, the system automatically utilizes existing annotation information within the clusters (such as sampled and labeled positive and negative samples) to train lightweight discriminant models such as SVMs (Support Vector Machines) to efficiently screen and sort images or samples. This method eliminates the need for full offline training and can dynamically adapt to the data distribution during queries, greatly improving query flexibility and accuracy. This automated discriminant model training based on a minimal number of annotated samples is another key innovation of this invention. It completely solves the problems of high annotation costs and slow model updates in the prior art, achieving truly low-cost, high-efficiency retrieval.

[0118] Supports multi-category combination queries (such as AND / OR / NOT set operations), and can flexibly combine multiple categories or keywords to return a collection of images that meet multiple conditions at the same time.

[0119] Search results can be sorted and filtered by multiple dimensions such as relevance, category, and confidence, further improving the flexibility and practicality of the search.

[0120] For example, in a product image retrieval system, when a user initiates a query for "red AND sneakers," the system first uses an inverted index to quickly locate clusters containing the categories "red" and "sneakers," narrowing the search from a total of one million images to approximately 20,000. The system then leverages a small number of existing labeled samples within these clusters (e.g., 5-10 labeled images of red sneakers per cluster) to automatically train a lightweight SVM model that accurately identifies images matching the "red sneakers" criteria. Finally, the system ranks the results by relevance, prioritizing images that best match the user's query intent. This entire process completes in milliseconds, providing users with an efficient and accurate search experience.

[0121] Aggregate query (aggregate): Specifically, users can specify categories, and the system uses inverted indexes to quickly count the number of samples and category distribution within related clusters, achieving efficient aggregate statistical analysis.

[0122] For example, in a video content analysis system, when content managers need to understand the overall status of "education" videos in the library, they can initiate an aggregate query. The system quickly locates all clusters containing the "education" label, summarizes the statistical information, and generates the following analysis report: "There are 1,250 education videos in total, with a total duration of approximately 185 hours. The main subcategories are: science experiments (35%), language learning (28%), math tutorials (20%), history explanations (12%), and other (5%). The average video duration is 8.9 minutes, and 215 new videos have been added in the past three months." This type of aggregate query can help users quickly obtain the macro characteristics of the dataset, supporting content planning and decision-making.

[0123] Limit query (limit): Specifically, the system supports users to limit the number or range of returned results. After sorting the search results, the system outputs a specified number of samples according to the threshold set by the user.

[0124] For example, in a text retrieval system, if a user needs to find high-quality documents related to "climate change," they can initiate a query with a restriction, such as "category=climate change AND confidence>0.8 LIMIT 20." The system will first find all documents related to climate change, then filter out high-quality documents with a confidence score greater than 0.8, and finally return the top 20 documents sorted by relevance. This restricted query ensures that users receive a selection of high-quality results, rather than a large number of potentially less relevant documents, effectively improving information acquisition efficiency.

[0125] The query processing process makes full use of indexes and clustering structures, first eliminating irrelevant data on a large scale, and then performing high-precision screening on a small scale, greatly improving retrieval efficiency and user experience.

[0126] Furthermore, the innovative introduction of online discriminant models (such as SVM) based on the annotation information within clusters enables efficient and intelligent automated screening, significantly outperforming traditional solutions that rely solely on static indexes. This design addresses the high cost of manual annotation in existing technologies by achieving high-quality query results with only a minimal number of annotated samples in each cluster, reducing annotation costs by an order of magnitude. Compared to traditional methods that require large amounts of annotated data, this method leverages the semantic consistency of cluster structures and requires only a minimal number of annotated samples (typically 5-10%) within each cluster to train a highly accurate discriminant model. This "few annotations, high accuracy" approach gives the present invention significant cost advantages and scalability in practical applications.

[0127] Among them, users do not need to understand the underlying index structure. The system automatically completes index screening and model judgment. The query experience is intelligent and efficient, which completely solves the problem that the existing technology's retrieval method is mainly based on keywords and tags, and it is difficult to achieve efficient retrieval based on content and semantics.

[0128] The overall architecture supports efficient management and real-time retrieval of large-scale data, and facilitates subsequent expansion and integration, effectively addressing the challenges of continuous growth and dynamic changes in unstructured data. Through semantically driven index construction and query processing, this invention achieves the transition from "data retrieval" to "knowledge discovery," providing users with a more intelligent, efficient, and user-friendly unstructured data management experience.

[0129] It should be noted that the above steps of the present invention can be flexibly configured and expanded according to actual application scenarios, and can adapt to the index construction and efficient retrieval requirements of different types of unstructured data.

[0130] An embodiment of the present invention provides a general unstructured data index construction device, which is used to implement a general unstructured data index construction method in the above embodiment. The device includes: The feature extraction module is used to preprocess the acquired raw unstructured data. For different types of unstructured data, it uses an adapted deep learning model to extract features and obtain high-dimensional feature vectors. The clustering grouping module is used to perform unsupervised grouping of high-dimensional feature vectors using a recursive clustering method based on automatic anchor point selection and hyperplane partitioning to form clusters with semantic consistency; Summary generation module, which uses a large language model to automatically generate a semantic summary for each cluster and associates the summary with the cluster; The index construction and query processing module is used to generate an independent index unit for each cluster, and realize the automatic screening of original unstructured data through the retrieval methods of selection query, multi-category combination query, aggregation query, and restriction query.

[0131] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0132] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0133] Correspondingly, an embodiment of the present invention further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the general unstructured data index construction method in the above embodiment is implemented.

[0134] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0135] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only.

[0136] The above embodiments are intended only to illustrate the design concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. The scope of protection of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design concepts disclosed in the present invention are within the scope of protection of the present invention.

Claims

1. A general method for constructing an unstructured data index, characterized in that: The following steps are involved: (1) Preprocess the acquired raw unstructured data and use an adapted deep learning model to extract features for different types of unstructured data to obtain high-dimensional feature vectors; (2) A recursive clustering method based on automatic anchor point selection and hyperplane partitioning is used to perform unsupervised grouping of high-dimensional feature vectors to form clusters with semantic consistency; (3) Using a large language model to automatically generate a semantic summary for each cluster and associate the summary with the cluster; (4) Generate an independent index unit for each cluster, and realize the automatic screening of original unstructured data through the retrieval methods of selection query, multi-category combination query, aggregation query, and restriction query.

2. The general unstructured data index construction method according to claim 1, characterized in that: In step (1), the unstructured data includes image, video and text data; for image data, a convolutional neural network is used to extract the visual features of the image and output a fixed-length high-dimensional feature vector to represent the content, structure and semantic information of the image; for video data, a temporal neural network or a multi-frame fusion model is used to extract the global temporal features of the video clip and output a feature vector that can reflect the dynamic content of the video; for text data, a large language model or an embedding model is used to extract the contextual semantic features of the text and output a feature vector that can reflect the text theme and semantic association; wherein all feature vectors are standardized.

3. The general unstructured data index construction method according to claim 1, characterized in that: Step (2) includes the following sub-steps: (2.1) In each round of cluster splitting, a representative semantic anchor point is selected from the samples of the current node as a candidate anchor point; the optimal anchor point is determined by calculating the similarity distribution between the candidate anchor point and other samples, taking into account the semantic consistency and distinguishability; (2.2) Using the optimal anchor point as a reference, calculate the similarity of all samples to the optimal anchor point and divide the samples into high-similarity and low-similarity groups. After removing outliers from the high-similarity and low-similarity groups, calculate the mean vector within each group and normalize the difference between the two mean vectors to obtain the main direction. Furthermore, generate multiple auxiliary directions orthogonal to the main direction and form a candidate direction pool by screening through orthogonality and dispersion. (2.3) For each candidate direction, traverse multiple split points and project the samples into two groups, the left and the right. Use the separability, intra-group consistency, and semantic discrimination indicators to evaluate each "direction + split point" combination, and select the "direction + split point" with the highest comprehensive score as the optimal hyperplane for the current node. (2.4) Taking the entire dataset as the root node, recursively perform anchor point selection, direction generation, optimal hyperplane selection, and splitting operations; Each split divides the sample into two child nodes, left and right, and processes them recursively until the termination condition is met; The recursive process forms a binary tree structure based on the hyperplane splitting basis, and each leaf node corresponds to a final cluster; (2.5) For all final clusters, classify them into different types based on the number of samples and internal consistency scores, identify abnormal nodes and optimize their clustering structure; (2.6) For each final cluster, count its semantic and mathematical features.

4. The general unstructured data index construction method according to claim 3, characterized in that: In sub-step (2.1), the selection of representative anchor points is based on the category distribution, density score, and semantic diversity of the samples, with priority given to samples that contain high-frequency categories, have high density, and strong diversity.

5. The general unstructured data index construction method according to claim 3, characterized in that: In sub-step (2.2), the cosine similarity between all samples in the current node and the selected anchor point is calculated, and the samples are sorted from high to low by similarity. Then, according to the configured sampling ratio parameter, a fixed proportion of samples are selected from the front and back ends of the sorting results to form high similarity groups and low similarity groups.

6. The general unstructured data index construction method according to claim 3, characterized in that: In sub-step (2.3), a quantile-based adaptive segmentation point selection strategy is used to project all samples in the candidate direction to obtain a one-dimensional projection value; multiple candidate segmentation points are uniformly sampled within the configured quantile range; for each candidate segmentation point, the samples are divided into two groups, the left and the right, based on whether the sample projection value is less than or equal to the segmentation point.

7. The general unstructured data index construction method according to claim 3, characterized in that: Step (3) includes: organizing the statistical information of the cluster into a structured summary prompt, which includes the main categories and their proportions, a brief description of representative samples, the total number of samples and category distribution; inputting the structured summary prompt into the large language model to automatically generate a natural language summary; and then associating the generated summary text with the unique identifier of the corresponding cluster.

8. The general unstructured data index construction method according to claim 1, characterized in that: Step (4) includes: generating an independent index unit for each cluster, including cluster ID, sample path, cluster statistical characteristics, semantic summary, main category distribution and annotation information; constructing an inverted index from category to cluster, supporting the location of related clusters by category; providing retrieval methods such as selection query, multi-category combination query, aggregation query, and restriction query to achieve automatic screening of original unstructured data; The selection query initiates a query through categories and keywords, uses the inverted index and clustering structure to exclude irrelevant data, searches only within the relevant clusters, narrows the candidate set, and uses the existing annotation information within the clusters to train the SVM model to screen and sort the pictures or samples within the candidate range; the multi-category combination query initiates a query by combining multiple categories or keywords, and returns a set of pictures that meet multiple conditions at the same time; the aggregation query initiates a query through a specified category, uses the inverted index to count the number of samples and category distribution within the relevant clusters, and realizes aggregate statistical analysis; the restriction query initiates a query by limiting the number or range of returned results, and after the search results are sorted, outputs a specified number of samples according to the set threshold.

9. A general unstructured data index construction device for implementing the method according to claim 1, characterized in that: include: The feature extraction module is used to preprocess the acquired raw unstructured data. For different types of unstructured data, it uses an adapted deep learning model to extract features and obtain high-dimensional feature vectors. The clustering grouping module is used to perform unsupervised grouping of high-dimensional feature vectors using a recursive clustering method based on automatic anchor point selection and hyperplane partitioning to form clusters with semantic consistency; Summary generation module, which uses a large language model to automatically generate a semantic summary for each cluster and associates the summary with the cluster; The index construction and query processing module is used to generate an independent index unit for each cluster, and realize the automatic screening of original unstructured data through the retrieval methods of selection query, multi-category combination query, aggregation query, and restriction query.

10. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by a processor, the universal unstructured data index construction method described in any one of claims 1 to 8 is implemented.

Images