A general method, apparatus and medium for constructing unstructured data indexes

Abstract

This invention discloses a general method, apparatus, and medium for constructing indexes for unstructured data, applicable to multimodal data such as images, videos, and text. The method includes: employing adapted deep learning models for feature extraction based on different modalities; constructing multi-level cluster structures using a recursive clustering method based on automatic anchor point selection and hyperplane partitioning; automatically generating semantic summaries for each cluster using a large language model; efficiently organizing the index structure through information such as category, cluster, and summary, supporting multi-dimensional retrieval; and during query processing, utilizing a minimal number of existing annotations within each cluster to train a discriminative model at low cost, achieving efficient cluster filtering and multi-category combination queries without requiring users to specify positive or negative samples. This method possesses advantages such as high automation, low annotation cost, strong scalability, support for multimodal data, and unique annotation inheritance, significantly improving the efficiency of index construction and retrieval for large-scale unstructured data.

Description

Technical Field

[0001] This invention relates to the fields of database management, artificial intelligence, information retrieval and large model applications, and in particular to a general method, apparatus and medium for constructing unstructured data indexes. Background Technology

[0002] With the rapid development of the Internet, the Internet of Things, and multimedia technologies, unstructured data (such as images, videos, text, and audio) is experiencing explosive growth across various industries. Statistics show that over 90% of newly generated data globally is unstructured. This data is widely found in scenarios such as social media, surveillance systems, medical images, corporate documents, and scientific research data, and is characterized by its large volume, diverse types, complex structure, and rich semantics.

[0003] However, traditional structured data indexing and retrieval methods (such as B+ trees, inverted indexes, and hash indexes) are mainly designed for tabular and relational databases, and are difficult to apply directly to high-dimensional, heterogeneous, and semantically complex unstructured data. Existing unstructured data management solutions typically have the following shortcomings:

[0004] Designed for a single modality or specific scenario, it lacks versatility and is difficult to adapt to the management and retrieval needs of various data types;

[0005] The search methods are mainly based on keywords and tags, which makes it difficult to achieve efficient retrieval based on content and semantics, and cannot meet the needs of intelligent applications;

[0006] The lack of an efficient index structure leads to low efficiency in large-scale data retrieval, high response latency, and difficulty in supporting real-time or near-real-time business scenarios.

[0007] Existing solutions rely heavily on manual or semi-automatic processes in index building, clustering, and summarizing, resulting in low levels of automation and intelligence, making it difficult to adapt to the continuous growth and dynamic changes in data.

[0008] With the development of deep learning and large-scale model technology, the capabilities of neural networks in feature extraction, clustering, and semantic understanding have been significantly improved, providing new ideas for the management and retrieval of unstructured data. However, how to systematically and automatically apply these capabilities to the indexing and retrieval of large-scale multimodal data still faces many challenges. Summary of the Invention

[0009] The purpose of this invention is to address the problems of low retrieval efficiency, poor index generalization, and high manual annotation costs of existing unstructured data (such as videos, images, and text). It proposes a general method, device, and medium for constructing unstructured data indexes, which enables efficient feature extraction, intelligent clustering, semantic summary generation, and automated query processing of multimodal data, thereby improving the management and retrieval capabilities of large-scale unstructured data.

[0010] The objective of this invention is achieved through the following technical solution:

[0011] A general method for constructing an index for unstructured data includes the following steps:

[0012] (1) Preprocess the acquired raw unstructured data. For different types of unstructured data, use an appropriate deep learning model to extract features and obtain high-dimensional feature vectors.

[0013] (2) A recursive clustering method based on automatic anchor point selection and hyperplane partitioning is adopted to perform unsupervised grouping of high-dimensional feature vectors to form clusters with semantic consistency;

[0014] (3) Use a large language model to automatically generate semantic summaries for each cluster and associate the summaries with the clusters;

[0015] (4) Generate an independent index unit for each cluster and achieve automated filtering of raw unstructured data through retrieval methods such as selection query, multi-category combination query, aggregation query, and restricted query.

[0016] Further, in step (1), the unstructured data includes image, video, and text data; for image data, a convolutional neural network is used to extract the visual features of the image and output a fixed-length high-dimensional feature vector to represent the content, structure, and semantic information of the image; for video data, a temporal neural network or a multi-frame fusion model is used to extract the global temporal features of the video segment and output a feature vector that can reflect the dynamic content of the video; for text data, a large language model or an embedding model is used to extract the contextual semantic features of the text and output a feature vector that can reflect the text topic and semantic association; wherein, all feature vectors are standardized.

[0017] Furthermore, step (2) includes the following sub-steps:

[0018] (2.1) In each round of cluster splitting, select representative semantic anchors from the samples of the current node as candidate anchors; by calculating the similarity distribution between the candidate anchors and other samples, and taking into account semantic consistency and discriminative ability, determine the optimal anchor.

[0019] (2.2) Using the optimal anchor point as a reference, calculate the similarity between all samples and the optimal anchor point, and divide the samples into high similarity group and low similarity group; after removing outliers from the high and low similarity groups respectively, calculate the mean vector within the group, and obtain the main direction by normalizing the difference vector between the two groups; further generate multiple auxiliary directions orthogonal to the main direction, and form a candidate direction pool by screening through orthogonality and dispersion.

[0020] (2.3) For each candidate direction, traverse multiple split points and project the samples into left and right groups; use separability, intragroup consistency and semantic discriminability indicators to evaluate each "direction + split point" combination, and select the "direction + split point" with the highest comprehensive score as the optimal hyperplane of the current node.

[0021] (2.4) With the entire dataset as the root node, recursively execute anchor point selection, direction generation, optimal hyperplane selection and splitting operations; each split divides the sample into two child nodes, left and right, and processes them recursively until the termination condition is met; the recursive process forms a binary tree structure based on the hyperplane as the splitting basis, and each leaf node corresponds to a final cluster.

[0022] (2.5) For all final clusters, classify them according to the number of samples and internal consistency scores, identify abnormal nodes and optimize their clustering structure;

[0023] (2.6) For each final cluster, calculate its semantic and mathematical features.

[0024] Furthermore, in sub-step (2.1), the selection of representative anchor points is based on the category distribution, density score and semantic diversity of the samples, and samples containing high-frequency categories, high density and strong diversity are preferred.

[0025] Further, in sub-step (2.2), the cosine similarity between all samples in the current node and the selected anchor point is calculated, the samples are sorted from high to low similarity, and then a fixed proportion of samples are selected from the front and back of the sorting results according to the configured sampling ratio parameter to form a high similarity group and a low similarity group.

[0026] Further, in sub-step (2.3), an adaptive segmentation point selection strategy based on quantiles is adopted to project all samples onto the candidate direction to obtain a one-dimensional projection value; multiple candidate segmentation points are uniformly sampled within the configured quantile range; for each candidate segmentation point, the samples are divided into left and right groups according to whether the sample projection value is less than or equal to the segmentation point.

[0027] Further, step (3) includes: organizing the statistical information of the clusters into a structured summary prompt, including the main categories and their proportions, a brief description of representative samples, the total number of samples and the category distribution; inputting the structured summary prompt into the large language model to automatically generate a natural language summary; and then associating the generated summary text with the unique identifier of the corresponding cluster.

[0028] Further, step (4) includes: generating an independent index unit for each cluster, including cluster ID, sample path, cluster statistical features, semantic summary, main category distribution and annotation information; constructing an inverted index from category to cluster to support locating related clusters by category; providing retrieval methods such as selection query, multi-category combination query, aggregation query and restricted query to realize automated filtering of raw unstructured data;

[0029] The selection query initiates a query by category and keywords, utilizes inverted indexes and clustering structures to exclude irrelevant data, and searches only within relevant clusters to narrow down the candidate set. Within the candidate set, an SVM model is trained using existing annotation information within the clusters to filter and sort images or samples. The multi-category combination query initiates a query by combining multiple categories or keywords, returning a set of images that simultaneously meet multiple conditions. The aggregation query initiates a query by specifying a category, utilizes inverted indexes to statistically analyze the number of samples and category distribution within relevant clusters, and achieves aggregate statistical analysis. The restriction query initiates a query by limiting the number or range of results returned, and after sorting the search results, outputs a specified number of samples according to a set threshold.

[0030] A general unstructured data indexing construction apparatus for implementing the above method includes:

[0031] The feature extraction module is used to preprocess the acquired raw unstructured data. For different types of unstructured data, an appropriate deep learning model is used to extract features and obtain high-dimensional feature vectors.

[0032] The clustering and grouping module is used to perform unsupervised grouping of high-dimensional feature vectors using a recursive clustering method based on automatic anchor point selection and hyperplane partitioning, forming clusters with semantic consistency.

[0033] The summary generation module is used to automatically generate semantic summaries for each cluster using a large language model and associate the summaries with the clusters;

[0034] The index building and query processing module is used to generate independent index units for each cluster. It enables automated filtering of raw unstructured data through retrieval methods such as selection query, multi-category combination query, aggregation query, and restricted query.

[0035] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described general method for constructing an unstructured data index.

[0036] The present invention has the following beneficial effects:

[0037] 1. The recursive clustering technique using automatic anchor point selection and hyperplane partitioning is more adaptable to the data distribution in high-dimensional feature space compared to traditional clustering methods such as K-means. It forms clusters with stronger semantic consistency, improves retrieval efficiency and accuracy, and solves the problem of slow retrieval speed of traditional indexing methods on large-scale unstructured datasets.

[0038] 2. The technique of automatically generating cluster summaries using a large language model provides richer semantic information compared to traditional indexing methods that rely solely on keywords or tags. This enhances the interpretability of search results and solves the problems of incomprehensible search results and lack of semantic relevance in existing technologies.

[0039] 3. By adopting a technique that combines a small number of annotations within clusters with online SVM training, the manual annotation cost is significantly reduced compared to traditional methods that require a large number of labeled samples, while maintaining good retrieval performance, thus solving the practical problem of high annotation costs for large-scale datasets.

[0040] 4. By adopting a unified multimodal processing framework, compared with indexing methods designed for a single modality, it can process different types of data such as images, videos and text in the same system, reducing system complexity and maintenance costs, and solving the problem of difficult multimodal data management;

[0041] 5. The modular system design ensures clear interfaces between components, facilitating future expansion and integration of new functions, enhancing system maintainability and scalability, and adapting to continuous data growth and dynamic changes. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort:

[0043] Figure 1 This is a schematic diagram of the hyperplane clustering process provided in an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram illustrating the summary generation and cluster association provided in an embodiment of the present invention;

[0045] Figure 3 This is a schematic diagram of the index structure and inverted index provided in an embodiment of the present invention;

[0046] Figure 4 This is a schematic diagram of the query processing flow provided in an embodiment of the present invention. Detailed Implementation

[0047] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.

[0048] This invention provides a general method for constructing unstructured data indexes, applicable to scenarios such as intelligent retrieval, content management, and data analysis. Specifically, it includes the following steps:

[0049] (1) Obtain the original unstructured data, preprocess and extract features from the original unstructured data; for different types of unstructured data, use an appropriate deep learning model to extract features, and transform the original unstructured data into high-dimensional feature vectors. Unlike existing technologies, this invention not only focuses on the surface features or simple mathematical representations of the data, but also delves deeper into the semantic connotations of the data, achieving true semantic understanding and retrieval. This semantic-based processing method is one of the core innovations of this invention that distinguishes it from existing technologies, enabling the system to understand the intrinsic meaning of the data, rather than relying solely on surface feature matching.

[0050] This invention can acquire raw unstructured data from multiple sources: for image data, it can come from enterprise product libraries, product images on e-commerce platforms, photos shared on social media, visual data collected by surveillance systems, etc.; for video data, it can come from online video platforms, educational and training resource libraries, corporate promotional materials, security monitoring systems, etc.; for text data, it can come from enterprise knowledge bases, news media reports, social media comments, customer feedback, technical documents, etc. For example, in retail industry applications, the system can process product images from product image libraries, promotional videos from marketing departments, and user review text from customer service systems, building a general index structure for this multimodal data and supporting multimodal intelligent retrieval services.

[0051] (1.1) Preprocessing of raw unstructured data:

[0052] Specifically, this step performs unified preprocessing operations to address the diversity and complexity of the original unstructured data, ensuring the effectiveness and consistency of subsequent feature extraction and clustering. Unlike existing methods designed for single modalities or specific scenarios, this invention provides a universal preprocessing framework capable of simultaneously processing various types of unstructured data such as images, videos, and text, thus solving the problem of lack of versatility in existing solutions.

[0053] Furthermore, for image data, we perform format unification (such as JPEG, PNG, etc.), size normalization, color space conversion, and noise reduction to eliminate differences in resolution, color, and noise between images from different sources; for video data, we perform frame extraction (such as equal-interval sampling or keyframe extraction), resolution unification, and frame rate standardization to ensure the temporal consistency and representativeness of video features; for text data, we perform word segmentation, stop word removal, unified encoding format, and removal of redundant and invalid characters to improve the processability and semantic purity of text content.

[0054] The preprocessing operations described above can be flexibly configured according to data type and actual needs to ensure that all data samples have a unified input standard before entering the feature extraction stage.

[0055] For example, in an enterprise media resource library containing multimodal data, the system needs to process heterogeneous data from different departments, devices, and time periods. For image data, the system converts various formats (BMP, TIFF, RAW, etc.) to JPEG format and standardizes the resolution to 1024×768 pixels to ensure consistency in subsequent feature extraction. For video data, the system extracts one keyframe per second from each video segment and standardizes the resolution of all videos to 720p and the frame rate to 30fps for subsequent processing. For text data, the system converts various formats (DOC, PDF, TXT, etc.) to plain text and performs Chinese and English word segmentation, removes stop words and special characters, and retains the core semantic content. Through this unified preprocessing framework, the system can transform the original data, which is of varying formats and quality, into standardized input, laying the foundation for subsequent feature extraction and index construction, and significantly improving the robustness and versatility of the retrieval system.

[0056] (1.2) Feature extraction:

[0057] Specifically, this step employs suitable deep learning models (such as convolutional neural networks, temporal neural networks, large language models, and embedding models) to extract features from different types of unstructured data, transforming the original unstructured data samples into high-dimensional feature vectors, i.e., multimodal feature vectors. This solves the problem that existing retrieval methods based on keywords and tags struggle to achieve efficient content- and semantic-based retrieval. The high-dimensional semantic feature vectors extracted by this invention through deep learning models can more comprehensively capture the semantic information of the data content, laying the foundation for subsequent intelligent retrieval. Unlike existing technologies that primarily rely on manually defined or shallow features, this invention utilizes high-dimensional features automatically learned and extracted by deep learning models, which can more accurately express the semantic connotation of the data, achieving a fundamental shift from "surface matching" to "semantic understanding."

[0058] Furthermore, for image data, convolutional neural networks (such as ResNet) are used to extract visual features of the images, outputting fixed-length high-dimensional feature vectors that can represent the content, structure, and semantic information of the images; for video data, temporal neural networks (such as 3D convolutional networks, temporal Transformers, etc.) or multi-frame fusion models are used to extract global temporal features of video segments, outputting feature vectors that can reflect the dynamic content of the video; for text data, large language models (such as BERT, etc.) or embedding models are used to extract contextual semantic features of the text, outputting feature vectors that can reflect the text's topic and semantic relationships.

[0059] All feature vectors undergo standardization (such as mean-variance normalization and L2 normalization) to ensure the accuracy of subsequent clustering and retrieval.

[0060] For example, in an intelligent image retrieval system, when a user uploads a query image containing a "beach sunset" scene, traditional keyword-based retrieval methods may only match images manually labeled "beach" or "sunset," failing to identify visually similar but incorrectly labeled images. This invention uses a ResNet-50 deep convolutional network to extract a 2048-dimensional feature vector, capable of capturing rich visual elements in the image, such as the orange-red sky, water reflections, and sand textures. Similarly, for video data, the system uses a 3D-CNN model to extract spatiotemporal features from a "street basketball game" video, not only identifying people and the basketball but also capturing action sequence features such as shooting and dribbling. For text data, the system uses a BERT model to transform an article on "AI ethics" into a 768-dimensional semantic vector. This vector not only encodes the article's topic but also includes its arguments, sentiment, and contextual relationships. This deep learning-based feature extraction method enables the system to understand the inherent semantics of the data, achieving truly content-based intelligent retrieval, significantly surpassing traditional keyword-matching retrieval methods.

[0061] (2) Hyperplane Clustering Grouping: A recursive clustering method based on automatic anchor point selection and hyperplane partitioning is adopted to efficiently group all high-dimensional feature vectors into clusters with good semantic consistency. This innovative design solves the problem of low efficiency and high response latency in large-scale data retrieval caused by the lack of efficient indexing structures in existing technologies. By constructing a multi-level semantic clustering cluster structure, the retrieval complexity is significantly reduced, providing support for real-time or near-real-time business scenarios. Unlike distance-based clustering methods such as K-means used in existing technologies, the hyperplane clustering method of this invention is more suitable for the characteristics of high-dimensional semantic spaces, can better capture the semantic structure of data, and form more interpretable clustering results. See also Figure 1The hyperplane clustering flowchart of this invention illustrates in detail the complete processing flow from the original feature vector to the final clustering result, which includes the following eight key steps:

[0062] The first step, "raw feature vectors (image / video / text features)," are high-dimensional feature vectors extracted from different modalities of data, which serve as input data for the clustering algorithm. These feature vectors contain semantic information from the image, video, or text data and form the basis for subsequent clustering analysis.

[0063] The second step, "preprocessing and normalization," standardizes the original feature vectors to eliminate scale differences between different feature dimensions, improving the accuracy and stability of subsequent clustering. This step ensures the comparability of feature vectors from different modalities.

[0064] The third step, "Semantic Anchor Selection (Representative Sample Sampling, Category / Density / Diversity Analysis)," selects the most representative samples from the current dataset as anchors. This step analyzes the category distribution, density score, and diversity characteristics of the samples to select sample points that best represent the characteristics of the data distribution, providing a reference benchmark for subsequent hyperplane partitioning.

[0065] The fourth step, "Hyperplane Direction Generation (Main Direction + Orthogonal Auxiliary Direction)," calculates the main direction that most effectively distinguishes the data based on the selected anchor points, and generates auxiliary directions orthogonal to the main direction. This design, combining main and auxiliary directions, enhances the hyperplane's partitioning capability and adaptability.

[0066] The fifth step, "Optimal Hyperplane Selection (Multi-index Scoring and Joint Optimization of Split Points)," evaluates the generated candidate hyperplanes from multiple dimensions, including indicators such as separability, intra-group consistency, and semantic discriminability. The best split point is determined through joint optimization to achieve the best data partitioning effect.

[0067] Step 6, "Recursive Splitting / Constructing Hyperplane Tree (Binary Tree Structure, Termination Condition Configurable)," recursively splits the dataset into two subsets (left and right) based on the selected hyperplane, forming a binary tree structure. This process continues until a preset termination condition is met, such as the number of node samples being less than a threshold, the tree depth reaching its upper limit, or the internal consistency of nodes being sufficiently high.

[0068] The seventh step, "Post-processing optimization (abnormal node correction, sample reassignment, and label attribution check)," optimizes the clustering results, including correcting abnormal nodes, reassigning boundary samples, and checking the consistency of label information attribution, in order to improve the quality and usability of the clustering results.

[0069] The eighth step, "Output Cluster Results", outputs the final clusters and their related information, which will serve as the basis for subsequent index construction and summary generation.

[0070] By organically combining these eight steps, this invention achieves efficient, interpretable, and semantically consistent clustering and grouping, laying a solid foundation for the construction of indexes for unstructured data.

[0071] For example, in a million-image retrieval scenario, traditional linear scanning methods require calculating the similarity between the query image and all images in the database one by one, and the retrieval time increases linearly with the amount of data. However, the hyperplane clustering method of this invention recursively divides the feature space into multiple semantically related subspaces, allowing retrieval to only access a few related clusters, reducing the retrieval complexity from O(n) to O(log n), achieving second-level response on million-image datasets. The specific process is as follows:

[0072] (2.1) Selection of semantic anchor points:

[0073] Specifically, during each round of cluster splitting, representative "semantic anchors" are first selected from the samples of the current node as candidate anchors. The selection of representative anchors is based on the class distribution, density score, and semantic diversity of the samples, with priority given to samples containing high-frequency classes, high density, and strong diversity.

[0074] By calculating the similarity distribution between candidate anchors and other samples, and taking into account both semantic consistency and discriminative ability, the optimal anchor is finally determined.

[0075] This step ensures that each split is centered on the most representative sample, improving the semantic interpretability of the clustering.

[0076] For example, in an image retrieval system, when clustering a dataset containing images of multiple categories, the semantic anchor selection algorithm of this invention prioritizes images located at the center of the category and rich in feature expression as anchors. For instance, in a cluster node containing the "car" category, the system selects car images with typical viewpoints, high clarity, and minimal background interference as anchors, rather than blurry, partially occluded, or unusually angled images, ensuring that subsequent hyperplane partitioning can maximize the preservation of the semantic integrity of the categories.

[0077] (2.2) Generation of hyperplane orientation:

[0078] Specifically, using the optimal anchor point as a reference, the similarity between all samples and the optimal anchor point is calculated, and the samples are divided into high similarity group and low similarity group. First, the cosine similarity between all samples in the current node and the selected anchor point is calculated; then, the samples are sorted from high to low similarity; next, according to configurable sampling ratio parameters (high_sim_ratio and low_sim_ratio), a fixed proportion (e.g., 15%) of samples are selected from the front and back of the sorting results, respectively, to form the initial high similarity group and low similarity group. This sorting-based partitioning method has stronger adaptability than traditional fixed threshold partitioning and can adapt to different data distribution characteristics.

[0079] Furthermore, after removing outliers from the high and low similarity groups respectively, the mean vector within the group is calculated, and the principal direction is obtained by normalizing the difference vector between the two groups' means.

[0080] To enhance splitting diversity and robustness, multiple auxiliary directions orthogonal to the main direction are generated, and a candidate direction pool is formed through orthogonality and dispersion screening.

[0081] This step ensures that the hyperplane direction of each split reflects both the main semantic distinction and diversity, avoiding getting trapped in local optima. This sophisticated outlier filtering strategy significantly improves the stability and semantic consistency of subsequent direction generation, representing a significant technological innovation in the hyperplane clustering direction generation stage of this invention.

[0082] For example, in a text retrieval scenario, when processing a collection of documents containing multiple topics, the hyperplane orientation generation algorithm of this invention first divides the documents into a high-similarity group (other sports news) and a low-similarity group (such as political and technological news) based on their similarity to the anchor document (such as a typical sports news article). Then, it calculates the semantic centers of the two groups to determine the main orientation that best distinguishes them. Simultaneously, the system also generates multiple auxiliary orientations, such as those distinguishing different sports (football vs. basketball). This multi-directional strategy enables the final clustering structure to capture multiple dimensions of text semantics, significantly improving retrieval accuracy.

[0083] (2.3) Optimal hyperplane selection:

[0084] Specifically, for each candidate direction, multiple split points are traversed, and the projected samples are divided into left and right groups. This invention employs a quantile-based adaptive split point selection strategy: First, all samples are projected onto the candidate directions to obtain one-dimensional projection values; then, within a configured quantile range (e.g., from the 10% quantile to the 90% quantile), multiple candidate split points (typically 20-50 points) are uniformly sampled; for each candidate split point, the samples are divided into left and right groups based on whether the projected sample value is less than or equal to that split point. This quantile-based partitioning method can adapt to different data distribution characteristics, avoid extreme splitting situations (such as excessively unbalanced partitioning), and ensure that both left and right groups have sufficient samples for subsequent evaluation.

[0085] Furthermore, this invention designs a semantically driven multi-dimensional scoring mechanism, using the comprehensive scoring formula Score(w)=γ·(μ) L -μ R )² / (σ L ²+σ R ²)+β·η+(1-γ-β)·δ, using multiple indicators such as separability, intragroup consistency, and semantic discriminability to evaluate each "direction + split point" combination. Among them, the separability term (μ L -μ R )² / (σ L ²+σ R ²) Measure the discriminative power of the projection of the left and right subclusters onto the direction w, μ L and μ R Let σ be the mean of the left and right groups in the projection direction, respectively. L ² and σ R ² represents the variance of the left and right groups, respectively; the internal consistency term η is evaluated by calculating the average cosine similarity between samples within the left and right child nodes and combining it with a sample size balancing factor; the semantic discrimination term δ evaluates the hyperplane's ability to distinguish different semantics by the similarity difference between the left and right groups and the anchor point, as well as the semantic difference between the left and right groups. γ, β, and (1-γ-β) are configurable weight parameters used to balance the importance of the three scoring dimensions.

[0086] Through multiple rounds of evaluation and sampling, the "direction + split point" with the highest comprehensive score is finally selected as the optimal hyperplane for the current node.

[0087] This step ensures that each split maximizes between-group differences, within-group consistency, and semantic interpretability.

[0088] The innovations of this invention in the hyperplane selection process include: (1) an adaptive segmentation point scanning strategy is adopted, which dynamically determines candidate segmentation points by configuring quantile ranges, thus avoiding suboptimal partitioning that may be caused by fixed segmentation points; (2) an efficient evaluation mechanism based on subsampling is introduced, which performs random subsampling on large-scale datasets before evaluation, significantly improving computational efficiency; (3) a semantically enhanced intra-group consistency evaluation method is designed, which not only considers the average similarity but also introduces the tightness of the similarity distribution as a supplementary indicator of clustering quality; (4) a multi-level semantic discrimination evaluation framework is developed, which ensures that the partitioning results have clear semantic interpretation by weighted combination of inter-group semantic differences and anchor point alignment differences. In actual execution, the system first performs a preliminary evaluation of all candidate directions and selects the directions with the highest scores; then, it performs a precise evaluation of these top candidate directions using the full dataset, and finally selects the "direction + segmentation point" combination with the highest comprehensive score as the optimal hyperplane.

[0089] For example, in a video retrieval system, when it is necessary to divide a collection of videos containing different content types, this invention evaluates multiple possible division schemes. For instance, the system may attempt to divide the video according to different semantic dimensions such as "indoor vs. outdoor," "people vs. scenery," and "static vs. dynamic," and calculate a score for each division scheme. Assuming that the "people vs. scenery" division scheme receives the highest score, the system will select this hyperplane as the splitting basis for the current node, thereby ensuring that the generated clusters have clear semantic boundaries, making it easier for users to quickly locate target videos through semantic descriptions.

[0090] (2.4) Recursively construct the hyperplane tree:

[0091] Specifically, taking the entire dataset as the root node, the anchor point selection, direction generation, optimal hyperplane selection, and splitting operations are recursively executed.

[0092] Furthermore, each split divides the sample into two child nodes, left and right, and processes them recursively until the termination condition is met (such as the number of node samples being less than a threshold, the tree depth reaching the upper limit, or high internal consistency within the node).

[0093] The recursive process forms a binary tree structure based on the hyperplane, with each leaf node corresponding to a final cluster.

[0094] This structure can adapt to data distribution and meticulously characterize the semantic structure in high-dimensional space.

[0095] For example, in a large-scale image retrieval system, this invention starts from a root node containing millions of images and constructs a multi-level hyperplane tree through recursive splitting. For instance, the first level of splitting might divide the images into two main categories: "natural scenes" and "artificial scenes"; the second level of splitting might further subdivide "natural scenes" into "landscapes" and "animals and plants," and "artificial scenes" into "buildings" and "transportation"; and so on, until leaf nodes with sufficiently high semantic consistency are formed. When a user queries images of the "mountain stream" category, the system only needs to quickly locate the relevant leaf node along the tree structure without scanning the entire dataset, thus achieving an order-of-magnitude improvement in retrieval efficiency.

[0096] (2.5) Post-processing optimization:

[0097] Specifically, for all leaf nodes (final clusters), the types are classified based on the number of samples and internal consistency scores to identify anomalous nodes. This invention designs a complete node classification and anomaly handling mechanism. First, a node classifier performs two-dimensional classification on all leaf nodes: based on the number of samples, they are divided into small, medium, and large nodes; based on consistency, they are divided into anomaly, good, and excellent nodes. The classification criteria are based on configurable threshold parameters, such as small_threshold_factor, medium_threshold_factor, and coherence_threshold, to ensure that the classification process can adapt to the characteristics of different datasets.

[0098] Furthermore, the clustering structure is optimized for anomalous nodes through sample reassignment and node merging to ensure high consistency within each cluster and reasonable assignment of anomalous samples. The anomalous node processor of this invention employs an iterative processing strategy, prioritizing small anomalous nodes, then medium-sized anomalous nodes, and finally large anomalous nodes. For small anomalous nodes, the system transfers all of their samples to the semantically most similar node; for medium-sized anomalous nodes, the system transfers a certain proportion (e.g., 20%) of their samples; and for large anomalous nodes, the system transfers a smaller proportion (e.g., 10%) of their samples. During the sample transfer process, the system prioritizes transferring samples with low semantic consistency with the overall node (outliers), and finds the optimal assigning node for each sample by calculating the similarity between the sample and the target node. In particular, this invention implements a synchronous migration mechanism for annotation information during the sample transfer process, ensuring that the annotation assignment of samples remains consistent and avoiding the loss or conflict of annotation information.

[0099] This step involves multiple rounds of iterative optimization until no abnormal nodes are found or the maximum number of iterations is reached, significantly improving the stability, consistency, and practical application effectiveness of the clustering results. The system calculates node consistency statistics before and after processing to quantitatively evaluate the optimization effect, providing a high-quality clustering foundation for subsequent index building and query processing.

[0100] For example, in text retrieval systems, some abnormal nodes may appear after clustering, such as "long-tail nodes" with too few samples or "mixed nodes" with low internal consistency. This invention optimizes these nodes, for example, by merging small clusters containing fewer than 10 documents into larger clusters with the most similar semantics, or by further splitting mixed clusters with significant differences in thematic content between documents. Through this post-processing optimization, the system can generate a more balanced and semantically pure clustering structure, improving the relevance of search results and user experience.

[0101] (2.6) Cluster feature statistics and annotation inheritance:

[0102] Specifically, for each final cluster, its semantic and mathematical features are statistically analyzed to facilitate subsequent summary generation and analysis.

[0103] Furthermore, semantic features primarily rely on a very small amount of manually labeled information. By extracting and labeling a very small number of samples from each cluster, the main category distribution and representative semantic description of that cluster can be obtained, greatly reducing the cost of manual labeling. This invention employs an adaptive sampling strategy, by default only extracting 10% of the samples in each cluster for labeling (minimum 10 samples). This proportion is sufficient to obtain accurate category distribution statistics in most scenarios. For example, for a cluster containing 500 samples, the system only needs to label 50 samples; while for small clusters (such as those with only 50 samples), the system will ensure that at least 10 samples are labeled to guarantee statistical reliability. In addition, this invention designs a label inheritance mechanism, prioritizing the use of existing labeling information and supplementing new labels only when necessary, further improving labeling efficiency.

[0104] Mathematical features reflect the distribution structure and diversity within a cluster by statistically analyzing the feature vectors of all samples within the cluster, including mean, standard deviation, and principal component analysis (PCA).

[0105] During clustering, the inheritance and migration of annotation information are supported, ensuring that the annotation of each sample is unique and complete. After operations such as sample reassignment and node merging, the annotation information is migrated synchronously with the sample, ensuring the integrity and consistency of the annotation data.

[0106] For example, in an image retrieval system, for a "cat" cluster containing 500 images, this invention only needs to randomly select 25 images (5%) for manual annotation to obtain the main category distribution of the cluster (e.g., "domestic cats" account for 60%, "wildcats" account for 30%, and "other felines" account for 10%). Simultaneously, the system calculates the mean, covariance, and other mathematical features of the feature vectors of these 500 images for subsequent similarity calculations and query processing. When some images in this cluster are reassigned to other clusters due to post-processing optimization, their annotation information is migrated accordingly, ensuring that the annotation status of each image in the system remains consistent and avoiding duplicate annotations and annotation conflicts.

[0107] This step achieves efficient, interpretable, and semantically consistent clustering in a high-dimensional feature space through mechanisms such as recursive hyperplane partitioning, anchor-driven direction generation and splitting, and post-processing optimization, laying a solid foundation for subsequent summary generation and index construction.

[0108] (3) Generation of large model summaries: See Figure 2 This invention utilizes generative artificial intelligence, such as large language models, to automatically generate semantic summaries for each cluster and associate these summaries with the clusters. This is one of the core innovations of the invention, organically combining large model technology with traditional indexing methods to achieve a high degree of intelligence and interpretability in the index structure. Compared with existing indexing methods that rely solely on keywords or tags, this invention innovatively leverages the powerful data analysis capabilities of large models to generate comprehensive semantic summaries for the data, accurately describing the data content and greatly improving the understandability of search results and user experience. The diagram illustrating the summary generation and cluster association of this invention details the complete process from multiple clusters to generating semantic summaries and establishing associations. This process includes the following six key steps:

[0109] The first step, shown in the dashed box at the top of the diagram, represents "multiple cluster families," including "Cluster Family 1 (sample path, feature vector, sampling label)," "Cluster Family 2," and so on up to "Cluster Family N." Each cluster family represents a set of semantically similar data samples formed by hyperplane clustering, which records the sample path, feature vector, and a small amount of sampling label information.

[0110] The second step involves compiling information from these clusters into the "Statistical Analysis (Vector Mathematical Features, Sampling Label Features)" module. Here, statistical analysis of mathematical and label features is performed on each cluster to provide an objective data foundation for subsequent summary generation.

[0111] The third step involves generating a "structured summary prompt" based on the statistical analysis results. This prompts the key information of the clusters into a structured template, which facilitates understanding and processing by the large language model.

[0112] The fourth step involves inputting structured prompts into the "large language model," which leverages its powerful language understanding and generation capabilities to perform semantic analysis and summarization of the cluster features.

[0113] In the fifth step, the large language model generates multiple semantic summary texts based on the structured prompts of the input, including "semantic summary text 1", "semantic summary text 2" up to "semantic summary text N". Each summary text describes the semantic features of the cluster from different perspectives.

[0114] The sixth step involves establishing a "summary and cluster ID association" by creating a unique mapping between the generated semantic summary text and the corresponding cluster, ensuring that each cluster has a corresponding semantic description, which facilitates subsequent index building and query processing.

[0115] By organically combining these six steps, this invention achieves high-quality semantic interpretation and expression of clustering results, greatly improving the intelligence and usability of unstructured data indexing.

[0116] Specifically, this step, for each cluster, automatically generates a summary text reflecting the main semantic features of the cluster using a large language model, improving the interpretability of the index and the semantic relevance of downstream searches. This innovative design solves the problem that existing technologies often involve manual or semi-automatic processes in index building, clustering, and summarizing, resulting in low levels of automation and intelligence. By introducing a large language model, it achieves highly automated summary generation, obtaining high-quality semantic descriptions without manual intervention, greatly enhancing the system's ability to process large-scale data. Compared with existing technologies, the large model summary generation of this invention not only significantly reduces manual costs but also provides richer and more accurate semantic descriptions, enabling non-professional users to easily understand and utilize search results. The specific process is as follows:

[0117] (3.1) Cluster statistical analysis:

[0118] Specifically, this step directly utilizes key information such as the main category distribution, representative samples, and sample size obtained in the previous step of cluster feature statistics, without the need for repeated calculations.

[0119] For example, in an image retrieval system, for a cluster containing landscape images, the system extracts the following statistical information: category distribution (e.g., "mountains" account for 45%, "beach" accounts for 30%, and "forest" accounts for 25%), representative samples (e.g., the 5 images closest to the cluster center), and total number of samples (e.g., the cluster contains 200 images). These objective statistical data provide a factual basis for subsequent summary generation, ensuring that the generated summary accurately reflects the content composition of the cluster, thereby supporting more precise retrieval.

[0120] These statistics provide a comprehensive and objective data foundation for summary generation, ensuring efficient workflow and data consistency.

[0121] (3.2) Abstract construction suggestions:

[0122] Specifically, the statistical information of the clusters is organized into a structured summary, which includes: the main categories and their proportions, a brief description of the representative samples, the total number of samples, and the category distribution.

[0123] Furthermore, the summary prompts can be organized using a template-based approach, such as "This cluster mainly includes category A (accounting for X%), category B (accounting for Y%), representative samples are..., and the total number of samples is N".

[0124] Structured prompts ensure that large language models can accurately understand the core semantics and statistical features of clusters.

[0125] For example, in a text retrieval system, for a cluster containing news articles, the system would organize the statistical information into the following structured prompt: "This cluster contains 120 news articles, mainly categorized as: sports news (65%), event reports (25%), and athlete interviews (10%). Representative article titles include 'Analysis of the 2023 World Cup Final' and 'Evaluation of Top Players' Season Performance.' Key keywords include: football, match, champion, player, tactics." This structured prompt enables the large language model to accurately grasp the core content and semantic features of the cluster, thereby generating a more precise summary.

[0126] (3.3) Large Language Model Summarization Generation:

[0127] Specifically, the structured summary prompts are input into the large language model, which automatically generates a natural language summary.

[0128] Furthermore, the summary generation process can be batch-processed to improve processing efficiency for large-scale datasets.

[0129] The generated summary should cover key information such as the main categories of the clusters, representative samples, sample quantity, and category distribution. The language should be concise, accurate, and readable.

[0130] If the generated abstract does not meet expectations, quality control can be performed by adjusting the prompts or adding constraints.

[0131] For example, in a video retrieval system, after inputting structured hints for a specific cluster into a large language model, the system might generate a summary like this: "This is a collection of 48 educational videos, mainly consisting of scientific experiment demonstrations (70%) and explanations of natural phenomena (30%). The videos primarily focus on high school physics and chemistry experiments, including detailed experimental procedures, explanations of principles, and results analysis. Most videos are between 5 and 10 minutes long, making them suitable as supplementary teaching materials." This natural language summary is richer and more specific than simple category labels, helping users quickly understand the characteristics of the cluster and improving the accuracy and efficiency of retrieval.

[0132] (3.4) Association storage of summaries and clusters:

[0133] Specifically, the generated summary text is associated with the unique identifier of the corresponding cluster to ensure that each cluster has an independent and traceable semantic description.

[0134] Abstract information, clustering results, and statistical information are included in the index structure to facilitate subsequent multi-dimensional retrieval and analysis.

[0135] For example, in an image retrieval system, the system assigns a unique identifier (e.g., "cluster_id_0042") to each cluster and establishes a persistent association between the generated summary (e.g., "This is a set of nighttime photos taken on city streets, mainly containing elements such as neon lights, traffic trails, and urban buildings. Most photos use long exposure technology to present a visual effect of flowing light and shadow") and this identifier. This association is stored in the system's index structure (e.g., a JSON-formatted index file). When a user searches for keywords such as "city nighttime view" or "neon lights," the system can quickly locate the relevant clusters and further verify their relevance through the summary information, thereby improving the accuracy of the retrieval.

[0136] This step, through meticulous statistical analysis, structured suggestion construction, automatic summary generation, and a rigorous summary-cluster association mechanism, achieves high-quality semantic interpretation and expression of clustering results, greatly enhancing the intelligence and practicality of unstructured data indexing.

[0137] (4) Index structure construction and query processing:

[0138] Specifically, this step focuses on clustering results and summary information to construct an efficient and scalable index structure and design an intelligent query processing flow, fully leveraging the system's engineering advantages. This invention comprehensively addresses several shortcomings in existing technologies through this design: it solves the lack of universality by supporting multimodal data processing; it addresses the difficulty in achieving efficient semantic retrieval through a content- and semantic-based index structure; it addresses the low efficiency of large-scale data retrieval through an efficient hierarchical index structure; and it addresses the high cost of manual intervention through an automated query processing flow, achieving a truly universal, efficient, and intelligent unstructured data indexing method. Compared to existing technologies, the index structure of this invention not only focuses on the physical organization of data but also emphasizes the expression and utilization of semantic relationships. Through the organic combination of clusters, summaries, and inverted indexes, it achieves semantically driven efficient retrieval, which is another core innovation that distinguishes this invention from existing technologies.

[0139] (4.1) Index structure construction: See Figure 3 The index structure and inverted index diagram of this invention illustrate in detail the organization of index units and their relationship with query processing. The diagram includes the following five key parts:

[0140] The first part, shown within the dashed box at the top of the diagram, represents the "cluster, summary index unit," which includes "cluster, summary 1 (ID, sample, summary)," "cluster, summary 2," and so on up to "cluster, summary N." Each index unit contains a unique identifier for the cluster, a list of sample paths, and its corresponding semantic summary; these are the basic building blocks of the index structure.

[0141] The second part extends from the index unit into three paths, connecting to "Category A Inverted Index (Category A → Cluster, Summary 1, 2)", "Category B Inverted Index (Category B → Cluster, Summary 1, N)" and "Category C Inverted Index (Category C → Cluster, Summary 2, N)". This indicates that the system constructs an inverted mapping relationship from categories to clusters, where a category can be associated with multiple clusters, and a cluster may also contain multiple categories.

[0142] The third part aggregates these inverted indexes into the "Query Request (e.g., Search by Category)" node, indicating that users can initiate queries by specifying categories, and the system will use the inverted indexes to quickly locate the relevant clusters.

[0143] In the fourth part, the query request is processed by "quickly filtering based on the summary to locate relevant clusters". The system uses semantic summary to perform preliminary screening of candidate clusters and quickly narrow down the search scope.

[0144] In the fifth part, the final system returns "cluster samples / summaries" as the query results, displaying data samples that meet the query conditions and their semantic descriptions to the user.

[0145] The system generates an independent index unit (such as a JSON file) for each cluster, containing cluster ID, sample path, cluster statistical features, semantic summary, main category distribution, and annotation information. Through this structured index organization, this invention achieves efficient data retrieval and management, enabling rapid response to user queries while maintaining good scalability.

[0146] Furthermore, an inverted index is constructed from categories to clusters to support quick location of relevant clusters by category, enabling efficient category retrieval and statistics.

[0147] For example, in an image retrieval system, for a dataset containing natural landscape photos, the system's index structure organizes semantically similar images into the same cluster. For instance, a cluster containing mountain scenery would have a unique identifier linking the path information of all images within that cluster. The system would generate a descriptive summary for this cluster, such as, "This is a set of mountain scenery photos taken at high altitudes, characterized by snow-capped peaks, blue skies, and expansive views. Most photos use wide-angle compositions, with blue and white as the dominant color tones." Simultaneously, the system would record the category distribution of images within the cluster (e.g., 65% of the images contain mountain elements, 25% contain lake elements, and 10% contain forest elements), as well as information on a small number of labeled samples and the mathematical characteristics of the cluster. This organization allows the system to quickly locate relevant clusters when a user queries "snow-capped mountain scenery," without needing to search all images individually.

[0148] All index data is kept synchronized with the clustering results to ensure data consistency and real-time performance.

[0149] The index structure is highly modular, making it easy to expand to support more search dimensions or integrate new functions in the future.

[0150] (4.2) Query processing flow: See Figure 4 The query processing flowchart of this invention illustrates in detail the complete process from user input to the return of query results. The flowchart includes three main query paths and multiple processing steps:

[0151] First, the top of the diagram is the "User Query Input" node, representing the starting point of a user's query. Based on the user's input type, the system divides the query request into three different processing paths, each optimized for specific query needs:

[0152] The first path is the "aggregate query" process:

[0153] 1. After a user enters an aggregation query request, the system first executes an "inverted index retrieval" to quickly locate the cluster containing the target category. This step utilizes a pre-built mapping relationship between categories and clusters, avoiding a full scan and significantly improving retrieval efficiency.

[0154] 2. Then, the system performs "summary content statistics (such as category distribution and quantity)" on the retrieved results. The system will analyze the summary information of the clusters, extract statistical features such as category distribution and object quantity, and perform summary calculations.

[0155] 3. Finally, the system returns aggregated statistical results to the user, including statistical information such as the total number of categories, the total number of objects, and the distribution ratio of categories, providing the user with an overall view of the dataset distribution and supporting data analysis and decision-making.

[0156] The second path is the "select query" process:

[0157] 1. After the user inputs their query request, the system executes "inverted index + preliminary summary screening (keywords / category / summary content)," combining the inverted index and summary information for initial screening. This step can quickly narrow down the search scope and improve the efficiency of subsequent processing.

[0158] 2. After initial screening, a "candidate cluster set" is obtained, containing clusters that may meet the query criteria. The system will extract sample paths and feature vectors from these clusters to prepare for subsequent precise screening.

[0159] 3. The system then uses the "intra-cluster SVM discriminant model (automatically trained with annotations)" to automatically train a support vector machine model based on a small number of labeled samples within the cluster. This step is one of the core innovations of this invention, as it enables the construction of an efficient discriminant model with a very small amount of labeled data, significantly reducing annotation costs.

[0160] 4. Then, perform "high-precision screening and sorting". Use the trained SVM model to accurately screen the candidate data and sort them according to the relevance score to ensure that the most relevant results are ranked first.

[0161] 5. Finally, "return precise search results" to the user, providing a high-quality result set that matches the query intent, greatly improving the user experience.

[0162] The third path is the "limit query" process:

[0163] 1. After the user inputs a restricted query request, the system performs "inverted index category screening" to quickly locate the clusters of relevant categories and establish a preliminary candidate set.

[0164] 2. Then apply the "limit condition (confidence level) filter" to filter the candidate set according to the confidence level threshold set by the user, and remove samples with insufficient confidence to ensure the quality of the results.

[0165] 3. Next, obtain the "most relevant candidate set", which contains the samples that best meet the criteria, in preparation for subsequent quantity limits.

[0166] 4. Then execute the "Select + Truncate / Limit Quantity" operation to limit the number of results to be truncated according to the number specified by the user, ensuring that the number of returned results meets the user's needs.

[0167] 5. Finally, return a limited number of results to the user, providing high-quality query results that meet the quantity limit.

[0168] These three query paths share the underlying feature vector management, cluster loading, and index structure, ensuring system consistency and efficiency. Through this design, the present invention can flexibly handle different types of query needs, providing efficient and accurate retrieval services, significantly improving user experience and system usability. The system offers users multiple efficient retrieval methods such as "selection query," "multi-category combination query," "aggregation query," and "restricted query," with its core advantages being:

[0169] Select query:

[0170] Specifically, users can initiate queries based on criteria such as category and keywords, without needing to understand the underlying index structure.

[0171] Furthermore, the system first utilizes inverted indexes and clustering structures to quickly eliminate a large amount of irrelevant data, and then performs subsequent searches only within the relevant clusters, significantly narrowing down the candidate set.

[0172] Within the candidate range, the system automatically utilizes existing annotation information within the clusters (such as sampled positive and negative samples) to train lightweight SVM (Support Vector Machine) and other discriminative models for efficient filtering and sorting of images or samples. This method does not require full offline training and can dynamically adapt to data distribution during querying, greatly improving the flexibility and accuracy of the query. This automated discriminative model training based on a very small number of labeled samples is another important innovation of this invention. It completely solves the problems of high annotation costs and slow model updates in existing technologies, achieving truly low-cost and high-efficiency retrieval.

[0173] It supports multi-category combination queries (such as AND / OR / NOT set operations), and can flexibly combine multiple categories or keywords to return a collection of images that meet multiple conditions at the same time.

[0174] Search results can be sorted and filtered according to multiple dimensions such as relevance, category, and confidence level, further enhancing the flexibility and practicality of the search.

[0175] For example, in a product image retrieval system, when a user initiates a search query for "red AND sneakers," the system first quickly locates clusters containing the categories "red" and "sneakers" using an inverted index, narrowing the search scope from all 1 million images to approximately 20,000. Then, the system automatically trains a lightweight SVM model using a small number of labeled samples already present in these clusters (e.g., 5-10 labeled red sneaker images in each cluster). This model accurately identifies images that match the "red sneakers" criterion. Finally, the system sorts the filtered results by relevance, placing the images that best match the user's query intent at the top. The entire process is completed within milliseconds, providing users with an efficient and accurate retrieval experience.

[0176] Aggregate query:

[0177] Specifically, users can specify categories, and the system can quickly count the number of samples and category distribution within related clusters through inverted indexes, achieving efficient aggregate statistical analysis.

[0178] For example, in a video content analysis system, when content managers need to understand the overall situation of "education" videos in the library, they can initiate an aggregate query. The system will quickly locate all clusters containing the "education" tag, summarize statistical information, and generate an analysis report like the following: "There are a total of 1250 educational videos, with a total duration of approximately 185 hours. The main subcategories are: science experiments (35%), language learning (28%), math tutorials (20%), history explanations (12%), and others (5%). The average video length is 8.9 minutes, and 215 new videos have been added in the last three months." This aggregate query helps users quickly obtain the macro-level characteristics of the dataset, providing support for content planning and decision-making.

[0179] Limit queries:

[0180] Specifically, the system supports users in limiting the number or range of returned results. After sorting the search results, the system outputs a specified number of samples according to the threshold set by the user.

[0181] For example, in a text retrieval system, when a user needs to find high-quality documents related to "climate change," they can initiate a query with restrictions, such as "Category = Climate Change AND Confidence > 0.8 LIMIT 20." The system will first find all documents related to climate change, then filter out high-quality documents with a confidence level greater than 0.8, and finally return the top 20 documents sorted by relevance. This restricted query ensures that users obtain carefully selected high-quality results, rather than a large number of potentially less relevant documents, effectively improving the efficiency of information retrieval for users.

[0182] The query processing workflow makes full use of indexes and clustering structures, first eliminating irrelevant data in a large range, and then performing high-precision filtering in a small range, which greatly improves retrieval efficiency and user experience.

[0183] Furthermore, this invention innovatively introduces an online discriminant model (such as SVM) based on annotation information within clusters, achieving efficient and intelligent automated filtering, significantly outperforming traditional schemes that rely solely on static indexes. This design addresses the high cost of manual annotation in existing technologies, achieving high-quality query results with only a very small number of labeled samples in each cluster, reducing annotation costs by an order of magnitude. Compared to traditional methods requiring large amounts of labeled data, this invention leverages the semantic consistency of cluster structures, requiring only a very small number of labeled samples (typically 5-10%) in each cluster to train a high-precision discriminant model. This "low-annotation, high-precision" characteristic gives this invention significant cost advantages and scalability in practical applications.

[0184] Users do not need to understand the underlying index structure. The system automatically completes index filtering and model discrimination, making the query experience intelligent and efficient. It completely solves the problem that existing retrieval methods are mainly based on keywords and tags, making it difficult to achieve efficient retrieval based on content and semantics.

[0185] The overall architecture supports efficient management and real-time retrieval of large-scale data, and is easy to expand and integrate, effectively addressing the challenges of the continuous growth and dynamic changes of unstructured data. This invention, through semantic-driven index building and query processing, achieves a leap from "data retrieval" to "knowledge discovery," providing users with a more intelligent, efficient, and user-friendly unstructured data management experience.

[0186] It should be noted that the above steps of the present invention can be flexibly configured and extended according to actual application scenarios, and can adapt to the indexing and efficient retrieval needs of different types of unstructured data.

[0187] This invention provides a general unstructured data index construction apparatus for implementing a general unstructured data index construction method described in the above embodiments. The apparatus includes:

[0188] The feature extraction module is used to preprocess the acquired raw unstructured data. For different types of unstructured data, an appropriate deep learning model is used to extract features and obtain high-dimensional feature vectors.

[0189] The clustering and grouping module is used to perform unsupervised grouping of high-dimensional feature vectors using a recursive clustering method based on automatic anchor point selection and hyperplane partitioning, forming clusters with semantic consistency.

[0190] The summary generation module is used to automatically generate semantic summaries for each cluster using a large language model and associate the summaries with the clusters;

[0191] The index building and query processing module is used to generate independent index units for each cluster. It enables automated filtering of raw unstructured data through retrieval methods such as selection query, multi-category combination query, aggregation query, and restricted query.

[0192] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0193] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0194] Accordingly, embodiments of the present invention also provide a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the general unstructured data index construction method described in the above embodiments.

[0195] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0196] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only.

[0197] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A general method for constructing an index for unstructured data, characterized in that, Includes the following steps: (1) Preprocess the acquired raw unstructured data. For different types of unstructured data, use an appropriate deep learning model to extract features and obtain high-dimensional feature vectors. (2) A recursive clustering method based on automatic anchor point selection and hyperplane partitioning is adopted to perform unsupervised grouping of high-dimensional feature vectors to form clusters with semantic consistency; (2.1) In each round of cluster splitting, select representative semantic anchors from the samples of the current node as candidate anchors; by calculating the similarity distribution between the candidate anchors and other samples, and taking into account semantic consistency and discriminative ability, determine the optimal anchor. (2.2) Using the optimal anchor point as a reference, calculate the similarity between all samples and the optimal anchor point, and divide the samples into high similarity group and low similarity group; after removing outliers from the high and low similarity groups respectively, calculate the mean vector within the group, and obtain the main direction by normalizing the difference vector between the two groups; further generate multiple auxiliary directions orthogonal to the main direction, and form a candidate direction pool by screening through orthogonality and dispersion. (2.3) For each candidate direction, traverse multiple split points and project the samples into left and right groups; use separability, intragroup consistency and semantic discriminability indicators to evaluate each "direction + split point" combination, and select the "direction + split point" with the highest comprehensive score as the optimal hyperplane of the current node. (2.4) Using the entire dataset as the root node, recursively execute anchor point selection, direction generation, optimal hyperplane selection, and splitting operations; Each split divides the sample into two child nodes, left and right, and processes them recursively until the termination condition is met. The recursive process forms a binary tree structure based on the hyperplane, with each leaf node corresponding to a final cluster; (2.5) For all final clusters, classify them according to the number of samples and internal consistency scores, identify abnormal nodes and optimize their clustering structure; (2.6) For each final cluster, calculate its semantic and mathematical features; (3) Use a large language model to automatically generate semantic summaries for each cluster and associate the summaries with the clusters; (4) Generate an independent index unit for each cluster and achieve automated filtering of raw unstructured data through retrieval methods such as selection query, multi-category combination query, aggregation query, and restricted query.

2. The general unstructured data index construction method according to claim 1, characterized in that, In step (1), the unstructured data includes image, video, and text data. For image data, a convolutional neural network is used to extract the visual features of the image and output a fixed-length high-dimensional feature vector to represent the content, structure, and semantic information of the image. For video data, a temporal neural network or a multi-frame fusion model is used to extract the global temporal features of the video segment and output a feature vector that reflects the dynamic content of the video. For text data, a large language model or an embedding model is used to extract the contextual semantic features of the text and output a feature vector that reflects the text topic and semantic association. All feature vectors are standardized.

3. The general unstructured data index construction method according to claim 1, characterized in that, In sub-step (2.1), the selection of representative anchor points is based on the class distribution, density score and semantic diversity of the samples, and samples containing high-frequency classes, high density and strong diversity are preferred.

4. The general unstructured data index construction method according to claim 1, characterized in that, In sub-step (2.2), the cosine similarity between all samples in the current node and the selected anchor point is calculated. The samples are sorted from high to low similarity. Then, according to the configured sampling ratio parameter, a fixed proportion of samples are selected from the front and back of the sorting results to form a high similarity group and a low similarity group.

5. The general method for constructing an unstructured data index according to claim 1, characterized in that, In sub-step (2.3), an adaptive segmentation point selection strategy based on quantiles is adopted to project all samples onto the candidate direction to obtain a one-dimensional projection value; multiple candidate segmentation points are uniformly sampled within the configured quantile range; for each candidate segmentation point, the samples are divided into left and right groups according to whether the sample projection value is less than or equal to the segmentation point.

6. The general unstructured data index construction method according to claim 1, characterized in that, Step (3) includes: organizing the statistical information of the clusters into a structured summary hint, including the main categories and their proportions, a brief description of the representative samples, the total number of samples and the category distribution; inputting the structured summary hint into the large language model to automatically generate a natural language summary; and then associating the generated summary text with the unique identifier of the corresponding cluster.

7. The general unstructured data index construction method according to claim 1, characterized in that, Step (4) includes: generating an independent index unit for each cluster, including cluster ID, sample path, cluster statistical features, semantic summary, main category distribution and annotation information; constructing an inverted index from category to cluster to support locating related clusters by category; providing retrieval methods such as selection query, multi-category combination query, aggregation query and restricted query to achieve automated filtering of raw unstructured data; The selection query initiates a query by category and keywords, utilizes inverted indexes and clustering structures to exclude irrelevant data, and searches only within relevant clusters to narrow down the candidate set. Within the candidate set, an SVM model is trained using existing annotation information within the clusters to filter and sort images or samples. The multi-category combination query initiates a query by combining multiple categories or keywords, returning a set of images that simultaneously meet multiple conditions. The aggregation query initiates a query by specifying a category, utilizes inverted indexes to statistically analyze the number of samples and category distribution within relevant clusters, and achieves aggregate statistical analysis. The restriction query initiates a query by limiting the number or range of results returned, and after sorting the search results, outputs a specified number of samples according to a set threshold.

8. A general unstructured data indexing construction apparatus for implementing the method as described in claim 1, characterized in that, include: The feature extraction module is used to preprocess the acquired raw unstructured data. For different types of unstructured data, an appropriate deep learning model is used to extract features and obtain high-dimensional feature vectors. The clustering and grouping module is used to perform unsupervised grouping of high-dimensional feature vectors using a recursive clustering method based on automatic anchor point selection and hyperplane partitioning, forming clusters with semantic consistency. The summary generation module is used to automatically generate semantic summaries for each cluster using a large language model and associate the summaries with the clusters; The index building and query processing module is used to generate independent index units for each cluster. It enables automated filtering of raw unstructured data through retrieval methods such as selection query, multi-category combination query, aggregation query, and restricted query.

9. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the general unstructured data index construction method as described in any one of claims 1 to 7.

Images