Method, device, computer equipment and medium for generating search results of server

Abstract

The present application relates to a method, apparatus, computer equipment and medium for generating retrieval results of a server, the method comprising: in response to a user's query request, indexing and retrieving the source data of the target server using a preset hierarchical index to determine the target data. Extracting the semantic vectors and visual features of the target data, and aligning the vectors and features according to preset alignment rules. Extracting key semantic units in the index retrieval path and generating a visual decision tree. Generating a natural language interpretation and a visual heat map based on the aligned semantic vectors and visual features. The visual decision tree, natural language interpretation and visual heat map constitute the retrieval results. The method combines multimodal feature extraction, feature alignment, visual decision tree, natural language interpretation and visual heat map generation technologies to achieve accurate generation and explainable display of server data retrieval results.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a method, apparatus, computer equipment, and medium for generating search results for a server. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, the demand for computing power among enterprises and university research teams is growing. AI training platforms, as a key tool to meet this demand, have as one of their core functions the ability to efficiently process information retrieval tasks across massive amounts of documents. Existing AI information retrieval systems mostly rely on static models trained offline, which are unable to adapt in real time to data distribution shifts and unexpected events. For example, when social media buzz or emerging technology terms rapidly emerge, static models lack data drift awareness and online adaptive update capabilities, making them incapable of meeting real-time requirements in scenarios such as news retrieval and financial risk control. Furthermore, traditional text retrieval techniques, such as TF-IDF (Term Frequency-Inverse Document Frequency) and BM25 (Best Matching 25), have significant limitations in cross-modal semantic matching for unstructured data such as images and videos. Existing cross-modal retrieval solutions typically employ independent feature extraction and simple vector concatenation, resulting in a significant semantic gap between the semantic spaces of text and images or videos. At the same time, due to the lack of efficient indexing mechanism for heterogeneous data (such as vectors of different dimensions), retrieval efficiency is also seriously affected. Summary of the Invention

[0003] To address these shortcomings, the present application provides a server-based search result generation method, apparatus, computer device, and medium. This method combines multimodal feature extraction, feature alignment, visual decision tree analysis, natural language interpretation, and visual heat map generation to achieve accurate generation and interpretable display of server data search results.

[0004] According to a first aspect, the present application provides a method for generating search results by a server, comprising:

[0005] In response to a user's query request, the target server's source data is indexed and retrieved using a preset hierarchical index to determine the target data that matches the query request. The hierarchical index is pre-built based on the source data. The semantic vectors and visual features of the target data are extracted and aligned according to preset alignment rules. Key semantic units in the data processing steps, feature matching process, and data flow path involved in the index retrieval are extracted to generate a visual decision tree. A natural language interpretation and a visual heat map are generated based on the aligned semantic vectors and visual features. The visual decision tree, natural language interpretation, and visual heat map are combined to form the retrieval results.

[0006] In some embodiments, before performing index retrieval on the source data of the target server using the preset hierarchical index, the method further includes:

[0007] The source data is obtained from the target server, and the source data includes text data and image or video data. Based on the application scenario of the target server and the set regulatory rules, the source data is processed for user privacy protection using differential privacy technology. The source data after user privacy protection processing is extracted using transfer learning technology to assist in identifying and processing outliers and missing values. The outliers and missing values ​​are labeled and verified using crowdsourcing annotation technology to obtain the verified source data. The semantic vectors of the text data and the visual features of the image or video data are extracted, and a hierarchical index is constructed based on the semantic vectors and visual features.

[0008] In some embodiments, extracting semantic vectors of text data and visual features of image or video data, and constructing a hierarchical index based on the semantic vectors and visual features includes:

[0009] A preset BERT-like model is used to extract semantic vectors from text data, and a preset convolutional neural network-like model is used to extract visual features from image or video data. The extracted text features and image features are aligned using preset contrast loss and cosine similarity loss functions, and the text and image feature spaces are aligned using a clipboard-like architecture. Text features are feature vectors representing the semantic information of text data, while image features are feature vectors representing the content of images or videos. The image feature space is the multidimensional space in which image features reside. A hierarchical inverted file index is constructed based on the aligned semantic vectors and visual features, which serves as the hierarchical index.

[0010] In some embodiments, when performing index retrieval on the source data of the target server using a preset hierarchical index, the method further includes:

[0011] Monitor the distribution shift of target data; when it is detected that the distribution shift exceeds the preset threshold, use the model-agnostic meta-learning framework to pre-train the preset index retrieval model, and construct a two-layer architecture of student model and teacher model as the architecture of the index retrieval model; among them, the student model is used for actual reasoning and prediction tasks, the teacher model is used to generate knowledge to guide the learning and optimization of the student model, and the index retrieval model is a model used to retrieve and process server data.

[0012] In some embodiments, when monitoring the distribution deviation of the target data, the method further includes:

[0013] The method includes maintaining the current mean and covariance matrix statistics of the target data using a mini-batch sampling calculation method, maintaining the current Wasserstein distance of the target data by setting a sliding window method, and setting a preset threshold and deviation value for the current Wasserstein distance; when new target data appears, updating the mean and covariance matrix statistics of the target data, and calculating the Wasserstein distance of the new target data; if the distribution deviation of the new target data is within a set range, reducing the preset threshold to a set first value; if the distribution deviation of the new target data exceeds the set range, increasing the preset threshold to a set second value; wherein the second value is greater than the first value.

[0014] In some embodiments, when constructing a two-layer architecture of a student model and a teacher model, the method further includes:

[0015] The dynamic distillation coefficient of the student model is set to exponentially decay with training time; the neural elastic weight consolidation and the ring buffer are set according to the preset regularization coefficient of the neural elastic weight consolidation and the preset capacity of the ring buffer; the dynamic data drift detection and model update process of the index retrieval model is set to a closed-loop feedback mode.

[0016] In some embodiments, the method further comprises:

[0017] The index retrieval model is compressed using dynamic channel pruning and / or quantization compression technology; the performance of the compressed index retrieval model is tested to obtain test results; and the pruning rate and / or quantization bit number of the index retrieval model are adjusted according to the test results.

[0018] According to a second aspect, the present application provides a retrieval result generating device of a server, the device comprising:

[0019] The target data retrieval module is used to, when receiving a user's query request, use a preset hierarchical index to index and retrieve the source data on the target server to determine the target data that matches the query request. The hierarchical index is pre-built based on the source data.

[0020] The vector feature extraction module is used to extract the semantic vector and visual features of the target data and align the semantic vector and visual features according to the preset alignment rules.

[0021] The decision tree generation module is used to extract the key semantic units in the data processing steps, feature matching process and data flow path involved in index retrieval, and generate a visual decision tree.

[0022] The heatmap generation module is used to generate natural language explanations and visualization heatmaps based on the aligned semantic vectors and visual features.

[0023] The retrieval result generation module is used to form retrieval results by visual decision trees, natural language explanations, and visual heat maps.

[0024] According to a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method for generating retrieval results of the server in any one of the above embodiments are implemented.

[0025] According to a fourth aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the computer device implements the steps of the server retrieval result generation method of any one of the above embodiments.

[0026] The server's retrieval result generation method can be applied to a cloud platform's intelligent index retrieval system (hereinafter referred to as the "system"). The method includes: responding to a user's query request, indexing and retrieving the target server's source data using a preset hierarchical index to determine target data that matches the query request. Extracting the semantic vectors and visual features of the target data, and aligning the semantic vectors and visual features according to preset alignment rules. Extracting key semantic units from the data processing steps, feature matching processes, and data flow paths involved in the index retrieval to generate a visual decision tree. Generating a natural language interpretation and a visual heat map based on the aligned semantic vectors and visual features. The visual decision tree, natural language interpretation, and visual heat map constitute the retrieval results. This method combines multimodal feature extraction, feature alignment, a visual decision tree, natural language interpretation, and visual heat map generation techniques to achieve accurate generation and interpretable display of server data retrieval results. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A flowchart of a method for generating search results by a server in one or more embodiments of the present application;

[0028] Figure 2 This is a flow chart of a method for preprocessing source data in one or more embodiments of the present application;

[0029] Figure 3 A flowchart of a method for extracting semantic vectors of text data and visual features of image or video data and constructing a hierarchical index in one or more embodiments of the present application;

[0030] Figure 4 A flowchart of a method for identifying and correcting abnormal text data in original user demand data in one or more embodiments of the present application;

[0031] Figure 5 This is a flow chart of a method for adaptively adjusting a preset threshold in one or more embodiments of the present application;

[0032] Figure 6 A flowchart of a method for finely controlling a student model learning process in one or more embodiments of the present application;

[0033] Figure 7 This is a flow chart of a method for compressing and optimizing an index retrieval model in one embodiment of the present application;

[0034] Figure 8 This is a schematic structural diagram of a retrieval result generating device of a server in another embodiment of the present application;

[0035] Figure 9 This is a schematic diagram of the internal structure of a computer device in one or more embodiments of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0037] According to a first aspect, this application provides a method for generating search results on a server. This method can be applied to an intelligent index retrieval system (hereinafter referred to as the "system") on a cloud platform. The system receives a query request from a user, then indexes and retrieves the source data of a target server based on the query request to obtain the target data. The system then generates a visual decision tree, natural language interpretation, and a visual heat map based on the target data to form the search results, and ultimately sends the search results to the user.

[0038] In certain exemplary embodiments of the present application, Figure 1 As shown, the method includes the following steps:

[0039] Step 101: In response to a user's query request, a preset hierarchical index is used to perform index retrieval on source data of a target server to determine target data that matches the query request.

[0040] Hierarchical indexing is pre-built based on source data. It indexes data according to a specific hierarchical structure, improving retrieval efficiency. The target server, which can be a single server or a server cluster, is used to store and manage source data. Source data includes text, images, or video data. Source data is a collection of raw information stored on the target server, awaiting retrieval. It encompasses various types of digital content, such as text in documents and visual content in images and video footage.

[0041] When constructing a hierarchical index, the system can divide the data into multiple levels for indexing based on the different characteristics and attributes of the data, such as first classifying by data type and then subdividing by content theme. For example, for server source data containing multiple media types (such as text, images, and videos), the system can first construct a text data layer index, an image data layer index, and a video data layer index. Each data layer index can then be further subdivided into index nodes based on content keywords, visual features, etc. When a query request is received, the system can quickly locate the corresponding data layer and index node based on the keywords or feature descriptions in the request, and then determine the target data that matches the query request.

[0042] For example, the specific process of index retrieval can be as follows: After receiving a user's query request, the system first preprocesses the query to extract key features from the query, such as semantic vectors for text queries or visual features for image queries. The system then matches these key features against a pre-built hierarchical index to identify the data blocks or data items most relevant to the query. The system then leverages the structural characteristics of the hierarchical index to quickly locate the data features that are closest to the query features through a multi-level indexing mechanism. Finally, at the text level, the system uses the Inverted File Flat (IVFFlat) structure for precise matching, and at the image level, the system uses the Inverted File Asymmetric Distance Computation with Product Quantization (IVFADCQ) structure for quantitative matching. By calculating the similarity between the query features and the feature vectors in the index, the system selects data features that are highly relevant to the query. Thus, the system begins with the user submitting a query request, and the features extracted after preprocessing enter the hierarchical index retrieval process. During index retrieval, data flows through the hierarchical structure of the tiered index, gradually refining from the top index to the bottom index, ultimately determining the data items that match the query request. The retrieved data items are returned to the user through the data flow path, forming a complete search path.

[0043] Step 102: extracting semantic vectors and visual features of the target data, and aligning the semantic vectors and visual features according to a preset alignment rule.

[0044] Semantic vectors are vectors used to represent the semantic information of text data. They can be extracted from text data using natural language processing models (such as BERT, Bidirectional Encoder Representations from Transformers). Visual features are features used to represent the visual information of image or video data. They can be extracted from text data using computer vision models (such as CNN, Convolutional Neural Network). Alignment rules are preset by the system and are used to map features of different modalities (such as text and images) into the same semantic space or to measure similarity, facilitating cross-modal matching and retrieval.

[0045] For text data, the system can use a BERT-like model to encode it into semantic vectors; for image or video data, the system can use a CNN-like model to extract its visual feature vectors. The preset alignment rule can use a contrastive loss + cosine similarity loss function (weight λ = 0.7). Through alignment training, the text semantic vector and the image visual feature vector have a similar distribution in the semantic space, thus achieving cross-modal feature alignment.

[0046] For example, for an image containing a "cat" and a text describing a "cat", after the system extracts and aligns them, the distance between the image's visual feature vector and the text's semantic vector in the semantic space will be closer, indicating that they have similar semantic information. In this way, during cross-modal retrieval, the text query can be effectively matched with the image data.

[0047] Step 103: Extract the key semantic units in the data processing steps, feature matching process and data flow path involved in index retrieval, and generate a visual decision tree.

[0048] Key semantic units refer to semantic information fragments or feature points that play a key role in determining target data during the retrieval process; a visual decision tree is a graphical tool that uses a tree structure to intuitively display the retrieval decision process and the relationship between key semantic units, helping users understand how the retrieval results are obtained.

[0049] During the indexing and retrieval process, the system records each step of data processing (such as data cleaning and feature extraction), specific feature matching operations (such as calculating similarity), and the data flow path within the system (e.g., the path from source data to index nodes and then to matching results). It then analyzes this recorded information using Concept Activation Vectors (CAVs), a technology used to identify and extract key semantic units. Key semantic units are then extracted and a visual decision tree is constructed based on their logical relationships and influence in the retrieval process.

[0050] For example, if the query is "Technology development trends in 2024," the system may process a large amount of technology-related document data during the retrieval process. After feature extraction and matching, it identifies key semantic units, such as "artificial intelligence," "cloud computing," "Internet of Things," and other related concepts, as well as their location and associations in different documents. The visual decision tree uses these key semantic units as nodes, connecting them according to their order and logical associations in the retrieval decision (for example, first determining whether the document topic is related to technology, then determining whether it involves development trends in 2024), to form an intuitive decision tree graphic, which is displayed to the user.

[0051] Step 104: Generate a natural language explanation and a visualization heat map based on the aligned semantic vectors and visual features.

[0052] Natural language explanation refers to explaining the search results in easy-to-understand natural language text to help users understand why such results are obtained; a visual heat map is a graphic that intuitively displays the distribution or importance of data features through different colors or brightness, and can be used to display information such as areas related to the query in images or videos.

[0053] The system uses Grad-CAM (Gradient-weighted Class Activation Mapping), a technique used to visualize the areas of interest of convolutional neural networks, to generate natural language explanations based on aligned semantic vectors and visual features, such as "Matching condition: {cat features with a similarity of 0.8 to the text description were detected in the image, and the location was in the center of the image}, with a confidence level of {0.2f}." Furthermore, the system uses Grad-CAM to generate heat maps of multimodal features. For image data, heat maps can highlight image regions relevant to the query. For example, in the aforementioned image of a cat, the heat map highlights the cat's outline, visually demonstrating how the model locates the key parts of the image based on the query.

[0054] For example, when a user queries for "landscape pictures with red flowers", the system generates a natural language explanation based on the aligned semantic vectors and visual features, which can be "The retrieved pictures contain an area with a high semantic similarity to 'red flowers', distributed in the upper left corner of the picture, with a similarity of 0.75". The visual heat map will highlight the flower area in the upper left corner of the picture, helping users quickly understand the match between the picture and the query request.

[0055] Step 105: The visual decision tree, natural language explanation, and visual heat map are combined to form the search results.

[0056] Search results are the result of the system integrating various forms of explanation and display content, ultimately presented to users, allowing them to fully and accurately understand the retrieved data and its relevance. The system can combine visual decision trees, natural language explanations, and visual heat maps in a specific layout and format to form a comprehensive search results page or document.

[0057] For example, the system's web interface displays a visual decision tree in the top half, graphically illustrating the search decision process. The middle section displays natural language explanations, detailing the matching criteria and confidence levels of the search results. The bottom section presents a visual heat map, visually displaying the distribution of key areas or features for image or video search results. This allows users to understand the search logic at a high level (via the decision tree) and gain a detailed understanding of the specific matches (via textual explanations and heat maps), thereby increasing their trust and understanding of the search results.

[0058] Through steps 101 to 105, the system can combine multimodal feature extraction, feature alignment, visual decision tree, natural language interpretation, and visual heat map generation technologies to achieve accurate generation and explainable display of server data retrieval results.

[0059] In some embodiments, as Figure 2 As shown, before step 101, the method further includes the following steps:

[0060] Step 201: Obtain source data from the target server.

[0061] The system accesses and reads various data files stored in the target server through a communication connection, extracts the specific content in these data files, and uses them as source data for subsequent processing.

[0062] For example, the target server may store a large number of news documents, user-uploaded images, and product introduction videos. The system can use a specific data reading interface or API (Application Programming Interface) to retrieve the text paragraphs in these documents, the binary data of image files, and the frame data of video files into the local processing environment in preparation for the next data processing operation.

[0063] Step 202: Based on the application scenario of the target server and the set regulatory rules, the source data is processed using differential privacy technology to protect user privacy.

[0064] The application scenario refers to the specific business usage environment of the target server, such as whether it is used for medical and health data management, e-commerce platform user information storage, or social platform content sharing; regulations and rules refer to the data privacy protection requirements and restrictions determined by relevant laws and regulations in the corresponding application scenarios, such as data confidentiality regulations in the medical industry and user privacy protection clauses in the Personal Information Protection Law; Differential privacy technology is a privacy protection technology that protects user privacy by adding noise to the data, making it impossible to accurately identify individual data in the data analysis results.

[0065] The system can first identify data that involves user privacy based on the application scenarios of different target servers. For example, in medical servers, patient names, diagnosis information, and treatment records are considered sensitive privacy data; in e-commerce servers, users' delivery addresses, contact information, purchase history, and other privacy-sensitive information must be protected. Then, based on the level of privacy protection stipulated by established regulations, differential privacy technology is used to add an appropriate amount of random noise to the corresponding data, while ensuring the overall effectiveness of data analysis. This makes it impossible to accurately restore each user's privacy details through external analysis results.

[0066] For example, on a social platform's target server, users' personal updates may contain private information such as their location information and daily life details. When the system utilizes differential privacy technology to process this data, it appropriately obscures the accuracy of the location information when storing it (for example, blurring street-level addresses to the district or county level). For text data such as daily life details, during statistical analysis (such as trending topic statistics), noise is added to make it impossible to determine whether a specific update originated from a specific user. This effectively protects users' personal privacy while ensuring that the social platform can properly perform overall data analysis functions such as trending topic statistics.

[0067] Step 203: Use transfer learning technology to extract features from the source data after user privacy protection processing to assist in identifying and processing outliers and missing values.

[0068] Transfer learning is a machine learning technique that applies knowledge and feature representations learned on one dataset or task to another related but different dataset or task, thereby leveraging existing knowledge to improve model training for new tasks, especially when the amount of new data is limited or labeling is difficult. Feature extraction refers to the process of automatically or manually obtaining key information from data that represents the core features of the data and is used for subsequent analysis and processing. Outliers refer to data points that significantly deviate from the normal range or expected patterns, which may be caused by data collection errors, entry errors, etc. Missing values ​​refer to situations where certain data items in a dataset have no values, which may be caused by reasons such as the data not being collected or being lost after collection.

[0069] Based on transfer learning technology, the system can first utilize models pre-trained on large-scale general datasets (such as the BERT model pre-trained on massive amounts of text data for text feature extraction, and the CNN model pre-trained on large amounts of image data for visual feature extraction). The system can then transfer the general feature representation knowledge learned by these pre-trained models to the source data processed for the current user's privacy protection. This approach allows for the rapid extraction of key features from the source data. For example, for text data, the semantic feature vector of the text can be extracted to represent its meaning and theme; for image or video data, the visual feature vector can be extracted to represent key visual information such as objects, scenes, and color distribution in the image or video. The system then uses the extracted feature information to compare and analyze the feature distribution of normal data to identify which data points in the source data are outliers (for example, in a set of temperature sensor data, if most of the data is between 20℃-30℃, but some data is 100℃, it may be an outlier). At the same time, it can also analyze the continuity and correlation of the features to find out where there may be missing values ​​(for example, in a continuous sequence of video frames, the visual features of some frames are obviously inconsistent with other frames, which may indicate that these frames are missing data).

[0070] For example, in a server containing user-written articles (text data) and user-taken landscape photos (image data), after extracting features using transfer learning technology, the system can obtain semantic theme feature vectors for each article in the text data. Next, when analyzing the distribution of these vectors, if the system finds that the semantic theme features of a particular article deviate significantly from those of other similar articles (for example, a technology article suddenly contains a large number of food-related semantic features), it can determine that the article data may contain outliers (perhaps due to a user mistakenly uploading incorrect content, or the text data was tampered with during storage, etc.). Furthermore, for image data, the visual feature vectors extracted by the system can be used to analyze the main objects and scenes in the photo. If the visual feature vector of a photo suddenly lacks the part representing the sky color, or the data at a location where a building outline should be located is abnormal, it can be determined that the image data may have missing values ​​or outliers caused by occlusion during shooting, storage errors, etc., which facilitates subsequent targeted processing of these outliers and missing values, such as correcting outliers and filling in missing values.

[0071] Step 204: Use crowdsourcing annotation technology to annotate and verify outliers and missing values ​​to obtain verified source data.

[0072] Crowdsourcing labeling technology refers to assigning the data labeling task to multiple participants (usually a large number of human labelers on the Internet) to complete, using the wisdom and knowledge of many people to accurately label and verify the data, thereby obtaining high-quality data labeling results. It is especially suitable for labeling tasks with large data volumes, high labeling difficulty, or those that require subjective judgment; labeling refers to adding labels to data, such as labeling object categories to images, labeling emotional tendencies to text, and other operations; verification processing refers to verifying the labeled data to ensure the accuracy and rationality of the labeling, so that the data is more in line with the requirements of subsequent processing.

[0073] The system uses transfer learning to extract and identify source data that may contain outliers and missing values. This data is then distributed to multiple crowdsourced annotators according to specific task allocation rules. For example, for outlier annotation tasks in text data, annotators can review the text content to determine whether it conforms to normal text logic and thematic requirements. Sections that clearly deviate from the topic or contain errors are labeled as outliers and provided with correction suggestions. For missing value annotation tasks in image data, annotators can review the images and mark areas with quality issues such as blur, occlusion, and missing content, as well as the possible types of missing content (e.g., "a portion of the top of the building is missing here"). After collecting annotation results from multiple annotators, the system can employ verification algorithms, such as majority voting (the majority opinion prevails), consistency checks (checking for consistency between annotators and further analyzing the reasons for any discrepancies), or comparative verification with expert annotation results, to verify the annotated data and select the most accurate and reasonable annotation information. This results in verified source data, ensuring data quality and reliability, and meeting the accuracy and effectiveness requirements of subsequent indexing and retrieval steps.

[0074] For example, when crowdsourcing annotation technology is used to process product review text and product display image data stored on an e-commerce server, crowdsourced annotators will mark obvious malicious and offensive language as outliers, or mark incomplete or unclear review text as missing values, and provide correction suggestions (such as completing the review content or rewording). For product display images, crowdsourced annotators will check whether the images fully and clearly display the product's appearance, features, and other information. Images with missing parts (such as missing product details) or excessive blur (making it difficult to clearly see the actual product) will be annotated with the corresponding missing or abnormal areas, and the missing part of the product will be indicated (such as "the front of the product box is missing"). After verification, the system filters out accurately annotated outliers and missing values, and corrects or supplements the data according to the annotation suggestions. Ultimately, the system obtains high-quality, verified source data, ensuring that subsequent retrieval steps can be based on accurate and complete data, improving the reliability of the entire retrieval system and the accuracy of search results.

[0075] Step 205: Extracting semantic vectors of text data and visual features of image or video data, and constructing a hierarchical index based on the semantic vectors and visual features.

[0076] A semantic vector refers to a vector form that can represent the semantic information of a text, which is converted into text data through a natural language processing model (such as BERT). It can map the meaning of the text to a point in a high-dimensional space, so that texts with similar semantics are closer in space; a visual feature refers to a feature vector extracted from image or video data through a computer vision model (such as CNN), which can represent its visual content information, including color, texture, shape, object category and other information; a hierarchical index is an index system constructed according to the different levels and structures of data features, which aims to improve the efficiency of index retrieval, so that when faced with large-scale data, it can quickly locate the data range related to the query request and reduce retrieval time.

[0077] For text data, the system uses a pre-trained BERT model, inputs the text into the model, and after processing through the model's multi-layer neural network, outputs a semantic vector corresponding to the text. For example, for a news report on "the development of artificial intelligence technology," the BERT model will comprehensively consider the vocabulary, context, and semantic relationships in the text to generate a fixed-dimensional semantic vector. This vector is close to the semantic vectors of other texts related to "artificial intelligence" in high-dimensional space, but is farther away from the semantic vectors of texts on unrelated topics such as "agricultural planting technology." For image or video data, a CNN model is used to perform operations such as convolution and pooling on the image to extract the image's visual feature vectors, such as the edges of objects in the image, color distribution, texture patterns, and other features. For video data, this can also include information such as motion features. Then, based on these semantic vectors and visual features, a hierarchical index is constructed.

[0078] For example, the system can first construct a top-level index based on data type (text, image, video), and then, within the index of each data type, further subdivide the index hierarchy based on features such as semantic themes (for text) or scene categories (for images and videos). For example, text data can be categorized according to themes such as science and technology, culture, and sports, and image data can be categorized according to scenes such as natural scenery, portraits, and architecture. Within each subdivided index hierarchy, more detailed index node divisions are performed based on the specific numerical ranges and similarities of semantic vectors or visual features, forming a multi-layered, structured index system. This enables efficient index retrieval, allowing the system to quickly locate the index node range where relevant text, image, or video data resides when processing user query requests, rapidly narrowing the search scope and improving retrieval efficiency.

[0079] Through steps 201 to 205, the system completes a series of preprocessing tasks, including source data acquisition, privacy protection, feature extraction, outlier and missing value processing, and hierarchical index construction. This lays a solid foundation for subsequent index retrieval steps, ensuring the high quality and accuracy of the data involved in the retrieval, and enabling rapid response to user query requests through an efficient indexing system.

[0080] In some embodiments, as Figure 3 As shown, step 205 includes the following steps:

[0081] Step 301: Using a preset BERT model to extract semantic vectors of text data, and using a preset convolutional neural network model to extract visual features of image or video data.

[0082] BERT-type models refer to natural language processing models based on the BERT architecture and its variants. They learn the contextual semantic information of text through pre-training and can convert text into vector representations with rich semantics. Convolutional neural network-type models refer to deep learning models based on CNN. They are good at processing image and video data, and extract visual features in the data, such as edges, textures, shapes, etc., through structures such as convolutional layers and pooling layers.

[0083] For text data, the system can perform pre-processing such as word segmentation and encoding on the text and then input it into a BERT model. The model will output a semantic vector corresponding to the text after multi-layer neural network operations. For image or video data, the system can perform pre-processing such as pixel normalization and cropping on the image or video frame and then input it into a convolutional neural network model. Local features will be extracted through convolution operations, and the feature dimensions will be reduced through the pooling layer to finally obtain a visual feature vector.

[0084] For example, for a science and technology news text, the system can use the BERT model to generate a semantic vector of length 768, which can reflect the semantic information such as scientific topics and key words involved in the text; for a natural landscape image, the system can use the convolutional neural network model to extract visual feature vectors containing elements such as mountains, rivers, and lakes. The vector dimension can be determined according to the model design, such as 2048 dimensions.

[0085] Step 302: align the extracted text features and image features using a preset contrast loss and cosine similarity loss function, and use a clipboard-like architecture to align the text features and image feature spaces.

[0086] Text features are feature vectors that represent the semantic information of text in text data, while image features are feature vectors that represent the content of images or videos in image or video data. The image feature space is the multidimensional space in which image features reside. Specifically, contrastive loss is a loss function used to measure the distance difference between positive and negative sample pairs. It can make positive sample pairs (semantically related text and images) closer in the feature space and negative sample pairs (semantically unrelated text and images) further apart. The cosine similarity loss function measures vector similarity based on the cosine value of the angle between vectors. The closer the value is to 1, the more similar the vectors are. This loss function aims to optimize the directional consistency of text features and image features, bringing semantically similar feature vectors closer in direction. The clipboard-like architecture is a model architecture that draws on the clipboard function. It can integrate and align feature information from different modalities (text and images) to construct a unified feature space.

[0087] The system can combine the extracted text features and image features into positive and negative sample pairs, using a contrastive loss function to calculate the distance difference loss between the positive and negative sample pairs. It also uses a cosine similarity loss function to calculate the directional similarity loss between the text and image features. These two loss functions are weighted and summed according to a preset weight (e.g., 0.7) to obtain a comprehensive loss. Backpropagation is then used to optimize model parameters to align text and image features in feature space. Furthermore, the system can employ a clipboard-like architecture to map text and image features into the same feature space, achieving unified representation of cross-modal features.

[0088] For example, in a dataset containing product description text and corresponding product images, the system can jointly optimize the contrast loss and cosine similarity loss functions to make the text features and image features describing the same product close in the feature space, while the features describing different products are far apart. Finally, through a clipboard-like architecture, the two are integrated into the same feature space, so that whether it is a text query or an image query, matching cross-modal data can be found in this space.

[0089] Step 303: constructing a hierarchical inverted file index based on the aligned semantic vectors and visual features to serve as a hierarchical index.

[0090] An inverted file index is an information retrieval index structure that enables fast retrieval by recording the document or data location corresponding to each feature vector. A hierarchical inverted file index organizes the inverted index according to a feature hierarchy, typically consisting of a coarse-grained clustering layer and a fine-grained inverted list layer. This improves retrieval efficiency on large datasets.

[0091] The system can first cluster the aligned semantic vectors and visual features to form multiple cluster centers and build a coarse-grained layer of the inverted file index; then, within each cluster, a fine-grained inverted list is established to record the data location corresponding to each feature vector (such as text paragraph location, image file location, etc.).

[0092] For example, for a server containing a large amount of text and image data, the system can first cluster the aligned semantic vectors and visual features into 1024 clusters to form a coarse-grained layer, where each cluster represents a semantic or visual theme; within each cluster, an inverted list is established to record in detail the specific data location corresponding to each feature vector in the cluster, such as the document ID and paragraph offset of text data, the file name and storage path of image data, etc.

[0093] Through steps 301 to 303, the system can extract and align features from text and image or video data, as well as efficiently construct an index. This provides a solid foundation for subsequent cross-modal data retrieval, enabling the system to quickly and accurately respond to complex user queries, improving the efficiency and accuracy of data retrieval and meeting user needs in multimodal data retrieval scenarios.

[0094] In some embodiments, as Figure 4 As shown, when executing step 101, the following steps are also included:

[0095] Step 401: Monitor the distribution deviation of target data.

[0096] Distribution shift refers to the phenomenon that there are significant differences between the statistical characteristics of the target data (such as the probability density function of the data distribution, characteristic mean, variance, etc.) and the statistical characteristics of the training data. This may be caused by changes in data over time (such as changes in user behavior, the generation of new content, etc.), changes in the external environment (such as seasonal factors, the impact of emergencies, etc.), or changes in the data collection method. This may lead to a decline in the performance of the index retrieval model and affect the accuracy and efficiency of the retrieval.

[0097] The system can monitor the distribution shift of target data by deploying a data drift perception module based on the Wasserstein distance. The Wasserstein distance is a metric that measures the difference between two probability distributions and can effectively reflect changes in the geometric characteristics of the data distribution.

[0098] For example, the system periodically extracts sample data from the target server and calculates its Wasserstein distance to the training data distribution. If this distance exceeds a preset threshold θ = 0.3, a distribution shift is detected. The threshold θ is set based on historical data and business requirements to balance sensitivity to distribution shift with the false positive rate.

[0099] Step 402: When it is detected that the distribution shift exceeds a preset threshold, a model-agnostic meta-learning framework is used to pre-train a preset index retrieval model, and a two-layer architecture of a student model and a teacher model is constructed as the architecture of the index retrieval model.

[0100] The student model is used for actual reasoning and prediction tasks, that is, it directly participates in responding to user queries and performing data retrieval; the teacher model is used to generate knowledge to guide the learning and optimization of the student model, and helps the student model better adapt to the new data distribution by providing additional supervision information (such as feature representation, similarity measurement, etc.); the index retrieval model is a model used to retrieve and process server data. Its core function is to quickly and accurately find query-related content from the massive data on the server based on the input query request.

[0101] When a distribution shift exceeding a threshold is detected, the system can initiate a pre-training process based on the MAML framework. The MAML framework optimizes the model's initial parameters, enabling it to quickly adapt to new tasks (i.e., retrieval tasks after distribution shift) on a small amount of new data. When constructing a two-layer architecture consisting of a student model and a teacher model, the teacher model generates guidance information based on its rich knowledge base (such as the feature representation capabilities accumulated during pre-training). For example, it uses an attention mechanism to highlight key feature areas that are helpful for the student model's learning or provides a reference standard for feature similarity measurement. The student model uses this guidance information, combined with new data samples, to quickly adjust its own parameters to adapt to the new data distribution.

[0102] For example, in a news retrieval scenario, when the system detects a sudden increase in user attention to emerging technologies (such as quantum computing), resulting in a shift in the distribution of related data, the teacher model can guide the student model to focus on the lexical features and semantic context related to "quantum computing", thereby enabling the student model to more efficiently retrieve the latest news content related to this emerging technology.

[0103] Through steps 401 to 402, the system can dynamically monitor and adapt to changes in the distribution of target data. By utilizing an advanced meta-learning framework and a two-layer model architecture, it ensures that the index retrieval model can maintain efficient and accurate retrieval performance even in scenarios where data distribution shifts, thereby improving the reliability and practicality of the entire server data retrieval system and meeting users' retrieval needs in a dynamic data environment.

[0104] In some embodiments, as Figure 5 As shown, when executing step 401, the following steps are also included:

[0105] Step 501: Maintain the current mean and covariance matrix statistics of the target data using a mini-batch sampling calculation method, maintain the current Wasserstein distance of the target data by setting a sliding window method, and set the preset threshold and deviation value for the current Wasserstein distance.

[0106] The mini-batch sampling calculation method is a method of randomly extracting a small number of samples from a large amount of data for calculation, which is used to efficiently estimate the statistical characteristics of the data; the mean and covariance matrix statistics are key indicators that describe the central trend of the data distribution and the correlation between features; the sliding window method is a data processing strategy that only considers the most recent period or a certain number of data samples to dynamically track data changes; the Wasserstein distance is used to measure the difference between two probability distributions and is used here to monitor changes in the target data distribution; the preset threshold is a judgment standard set according to business needs and historical data, which is used to determine whether the distribution deviation exceeds the normal range; the deviation value is used to measure the fluctuation range of the Wasserstein distance to assist in determining the stability of the distribution change.

[0107] The system can periodically extract small batches of samples from the target data (e.g., 100 samples at a time) and calculate and update the mean and covariance matrix of these samples to estimate the overall statistical characteristics of the target data. Simultaneously, a fixed-size sliding window (e.g., containing the most recently calculated Wasserstein distance values) is set to maintain the current Wasserstein distance of the target data. Based on historical data distribution and business requirements, the system sets a preset threshold (e.g., θ = 0.3) and deviation value (e.g., δ = 0.05) for the Wasserstein distance to facilitate subsequent distribution shift assessment.

[0108] For example, in a financial transaction data retrieval system, the target data is real-time transaction records. Every minute, the system extracts the 100 most recent transaction records as a small batch sample and calculates their mean (e.g., average transaction amount) and covariance matrix (e.g., correlation between different transaction types). Simultaneously, a sliding window containing the 10 most recently calculated Wasserstein distance values ​​is maintained. After each new Wasserstein distance calculation, the window is updated and the oldest data is removed. Based on the distribution of historical transaction data and the stability requirements of financial services, the Wasserstein distance is set to a preset threshold of 0.3 and a deviation of 0.05. This means that when the Wasserstein distance exceeds the range of 0.3±0.05, a significant distribution shift is considered to have occurred.

[0109] Step 502: When new target data appears, the mean and covariance matrix statistics of the target data are updated, and the Wasserstein distance of the new target data is calculated.

[0110] New target data refers to the latest data continuously generated during system operation. This data may reflect changes in user behavior, market dynamics, or other environmental factors. Updating the mean and covariance matrix statistics refers to adjusting previous statistical estimates based on the new data to reflect the latest changes in the data distribution. Calculating the new Wasserstein distance compares the new data distribution with the reference data distribution (usually the initial training data distribution) to obtain a numerical metric that measures the difference in data distribution.

[0111] Whenever new target data enters the system, it adds it to the current mini-batch and recalculates the mean and covariance matrix statistics. The system then uses the updated statistics and the reference data statistics to calculate the new Wasserstein distance using the Wasserstein distance formula. This new distance is then added to the sliding window, while the oldest distance value in the window is removed.

[0112] For example, in the aforementioned financial transaction data retrieval system, when a new transaction record is generated, the system adds it to the current mini-batch and recalculates the mean and covariance matrix. Suppose the newly added transaction record reveals a new trading pattern (e.g., a large amount of high-frequency trading), which would result in a change in the mean and covariance matrix. The system then calculates a new Wasserstein distance. Suppose this distance is 0.35, exceeding the preset threshold range (0.3 ± 0.05), indicating a significant shift in the distribution of the target data.

[0113] Step 503: If the distribution deviation of the new target data is within the set range, the preset threshold is reduced to a set first value; or if the distribution deviation of the new target data exceeds the set range, the preset threshold is increased to a set second value.

[0114] The second value is greater than the first value. Distribution deviation refers to the degree of difference between the Wasserstein distance of the new target data and the Wasserstein distance of the reference data, and is used to measure the magnitude of change in the data distribution. The set range is an allowable variation interval predetermined based on business needs and data stability, and is used to determine whether the distribution deviation is within an acceptable range. The adjustment of the preset threshold is to dynamically adapt to changes in data distribution and improve the robustness of the system to different data environments. The first and second values ​​are threshold adjustment ranges pre-set according to the system design. The second value is greater than the first value to ensure that the system can more sensitively detect abnormal changes when the distribution deviation is large.

[0115] When the Wasserstein distance of new target data falls within the range of the preset threshold ± the deviation value, the system considers the data distribution change to be within an acceptable range and reduces the preset threshold to a first value (e.g., θ1 = 0.25) to improve the detection of subtle distribution changes. Conversely, when the Wasserstein distance of new target data exceeds this range, the system considers the data distribution to have changed significantly and increases the preset threshold to a second value (e.g., θ2 = 0.35) to avoid frequent false alarms caused by oversensitivity.

[0116] For example, in a financial transaction data retrieval system, if the new Wasserstein distance is 0.28, which is within the preset threshold range of 0.3±0.05 (i.e., between 0.25 and 0.35), the system will reduce the preset threshold to 0.25 to more accurately monitor subsequent data changes. If the new Wasserstein distance is 0.40, which is outside this range, the system will increase the preset threshold to 0.35 to accommodate larger data distribution changes and avoid frequent distribution shift alerts caused by too low a threshold.

[0117] Through steps 501 to 503, the system can dynamically monitor the distribution changes of target data and adaptively adjust the preset thresholds, thereby reducing false positives and missed positives while ensuring sensitivity to data distribution deviations.

[0118] In some embodiments, as Figure 6 As shown, when executing step 402, the following steps are also included:

[0119] Step 601: Set the dynamic distillation coefficient of the student model to decay exponentially with training time.

[0120] The dynamic distillation coefficient refers to the parameter that controls the intensity of knowledge learned by the student model from the teacher model during the knowledge distillation process. This parameter will be adjusted according to the training progress to balance the student model's learning emphasis on historical knowledge and new knowledge.

[0121] The dynamic distillation coefficient can be set using the following formula:

[0122] ;

[0123] α represents the dynamic distillation coefficient, t represents the training time, and β is a hyperparameter that controls the decay rate.

[0124] For example, at the beginning of training, the dynamic distillation coefficient α is close to 1, and the student model mainly learns the knowledge provided by the teacher model; as the training time t increases, α gradually decays exponentially, and the student model gradually focuses on learning new knowledge, thereby achieving a smooth transition from relying on historical knowledge to being dominated by new knowledge.

[0125] Step 602: Setting the neural elasticity weight consolidation and the ring buffer according to the preset regularization coefficient of the neural elasticity weight consolidation and the preset capacity of the ring buffer.

[0126] Elastic Weight Consolidation (EWC) is a regularization method used to prevent catastrophic forgetting in models. It retains key knowledge already learned by penalizing large changes to important parameters. The regularization coefficient controls the strength of EWC, determining the degree to which the model retains old knowledge. A ring buffer is a storage structure used to store key data or parameters during model training. When the buffer is full, new data overwrites the oldest data, utilizing storage space in a circular manner.

[0127] The default regularization coefficient λ is set to e 3 , used to measure the model's retention strength of old knowledge; the capacity N of the ring buffer is set to e 5 , used to store the key historical parameters of the model.

[0128] For example, during the model update process, EWC calculates the Fisher information matrix of important parameters and combines it with the regularization coefficient λ to limit the update range of these parameters to prevent the model from forgetting important knowledge. 5 The feature representation of each training sample is used to provide historical information reference when optimizing the model.

[0129] Step 603: Set the dynamic data drift detection and model update process of the index retrieval model to a closed-loop feedback mode.

[0130] The closed-loop feedback mode refers to the direct use of dynamic data drift detection results to trigger model updates, and the performance of the updated model is fed back to the detection link, forming an automatically adjusted cycle to ensure that the model always adapts to the latest data distribution.

[0131] When the data drift perception module detects a distribution shift exceeding a preset threshold, it triggers a model update process, including retraining the student model and adjusting teacher model parameters. The updated model's performance (such as retrieval accuracy and response time) is monitored and fed back to the data drift detection process, enabling dynamic adjustments to subsequent detection sensitivity and threshold settings.

[0132] For example, in a news retrieval system, when the system detects that user interest in emerging technologies has caused a shift in data distribution, it automatically updates the index retrieval model. After the update, the system appropriately relaxes the data drift detection threshold based on the model's improved retrieval accuracy, reducing unnecessary update triggers while ensuring that the model can promptly adapt to new data changes.

[0133] Through steps 601 to 603, the system can achieve fine control over the student model learning process, effectively preventing catastrophic forgetting of the model, and at the same time utilize a closed-loop feedback mechanism to ensure the continuous optimization and efficient operation of the index retrieval model in a dynamic data environment.

[0134] In some embodiments, as Figure 7 As shown, the following steps are also included:

[0135] Step 701: compress the index retrieval model using dynamic channel pruning and / or quantization compression technology.

[0136] Dynamic channel pruning is a model compression technology that reduces the model's computational workload and storage requirements by dynamically removing neural network channels that have little impact on model output during model training or inference. Quantization compression technology reduces the precision of model weights (for example, quantizing 32-bit floating-point numbers to 8-bit integers) to further reduce the model's storage space and computational complexity while maintaining model performance as much as possible.

[0137] For dynamic channel pruning, the system can monitor the contribution of each channel in real time during training (for example, by calculating the L1 norm of the channel's activation value to measure its importance) and prune (i.e., remove) channels whose contribution falls below a preset threshold. The pruning rate (i.e., the proportion of channels pruned) can be dynamically adjusted based on the complexity and performance requirements of the model. For example, the initial pruning rate is 40%, and then gradually increased to 60% based on model performance. For quantization compression, the system can adopt a progressive quantization strategy, first quantizing the model weights from 32-bit floating point numbers to 16-bit floating point numbers, and then further quantizing them to 8-bit integers. At the same time, quantization-aware training (QAT) is used during the quantization process to compensate for the accuracy loss caused by quantization.

[0138] For example, in a CNN-based image retrieval model, the original model had 64 channels, each containing 128 neurons. Through dynamic channel pruning, the system discovered during training that 16 of these channels had low L1 norms for activation values, which had little impact on the model output. These channels were then pruned, reducing the model's channel count to 48 and computational complexity by approximately 25%. Furthermore, quantization compression technology was used to quantize the model weights from 32-bit floating point numbers to 8-bit integers, reducing the model's storage space by 75% (from 32 bits to 8 bits). Furthermore, through quantization-aware training, the model's retrieval accuracy only decreased by 1.2%.

[0139] Step 702: Perform a performance test on the compressed index retrieval model to obtain a test result.

[0140] Performance testing involves evaluating a model across multiple key metrics, including but not limited to search accuracy (such as Top-1 and Top-5 accuracy), inference latency (the time it takes the model to process a single query), storage usage, and computing resource consumption. Test results are used to guide subsequent model optimization efforts.

[0141] The system can evaluate the compressed model on multiple test sets covering different data distributions and query types to fully reflect the model's actual performance. For example, on a test set containing 10,000 images, the model's retrieval results for each image are tested to see if they correctly match the corresponding text description, and the processing time for each query is recorded. Simultaneously, model analysis tools (such as NVIDIA Nsight Systems) are used to monitor the model's computing resource usage on the CPU (Central Processing Unit) and GPU (Graphics Processing Unit).

[0142] For example, test results show that the compressed model achieved a Top-1 accuracy of 82.3% and a Top-5 accuracy of 94.7%. The average inference latency for a single query was 120 milliseconds, the model storage space was reduced from the original 128MB to 32MB, the CPU utilization was reduced from 85% to 60%, and the GPU utilization was reduced from 90% to 65%. These results demonstrate that the model maintains high retrieval performance after compression while significantly reducing resource consumption.

[0143] Step 703: Adjust the pruning rate and / or quantization bit number of the index retrieval model according to the test results.

[0144] The pruning rate refers to the ratio of pruned channels to the total number of channels, and the quantization bit count refers to the number of bits used after quantization of the model weights (for example, quantizing from 32-bit floating point numbers to 8-bit integers uses 8 bits of quantization). The goal of adjusting the pruning rate and quantization bit count is to achieve the best balance between model performance and compression rate.

[0145] If test results show that the model's retrieval accuracy has dropped significantly, the system can appropriately reduce the pruning rate (for example, from 60% to 40%) or increase the number of quantization bits (for example, from 8 bits to 16 bits) to mitigate the accuracy loss caused by compression. Conversely, if the model's performance remains significantly higher than expected, the pruning rate can be further increased or the number of quantization bits can be reduced to further reduce the model's storage space and computational complexity.

[0146] For example, based on the test results above, if the business scenario requires a Top-1 accuracy of at least 85%, and the current model's Top-1 accuracy is 82.3%, which is lower than the expected requirement, the pruning rate can be reduced from 60% to 50%, while the number of quantization bits can be increased from 8 to 16. After retraining and retesting, the model's Top-1 accuracy improves to 86.5%, while the inference latency increases to 150 milliseconds and the storage space increases to 48 MB. These improvements still significantly outperform the original model in terms of performance and resource usage. Through this iterative optimization, the system ultimately determined a pruning rate of 50% and a quantization bit size of 16 as the optimal configuration for model compression.

[0147] Through steps 701 to 703, the system can effectively compress and optimize the index retrieval model, significantly reducing storage space and computing resource consumption while ensuring that the model retrieval performance meets business needs, and improving the deployment efficiency and operation speed of the model.

[0148] In one embodiment of the present application, the server's retrieval result generation method can be applied to an intelligent index retrieval system (hereinafter referred to as the "system") of a SaaS (Software as a Service) platform. When a user sends a query request through the SaaS platform, the system first responds to the request and uses a preset hierarchical index to index and retrieve the source data of the target server to determine the target data that matches the query request. The hierarchical index here is an efficient index structure that can simultaneously process multiple types of data such as text and images, and improve the accuracy and speed of retrieval by constructing multiple index layers. For example, the system may use a combination of an inverted index based on text content and a vector index based on image features to quickly locate data items that match the text keywords and image features in the user's query.

[0149] After determining the target data, the system further extracts semantic vectors and visual features from this data. Semantic vectors are generated using natural language processing techniques (such as the BERT model) and can capture the deep semantic information of text data. Visual features, on the other hand, are extracted from image data using computer vision techniques (such as the CNN model) and can represent the primary visual content of the image. The system aligns these semantic vectors and visual features according to preset alignment rules to ensure that data from different modalities can be effectively compared and integrated within the same semantic space. For example, the system may use a combination of contrast loss and cosine similarity loss to minimize the differences between text and images in the semantic space, thereby achieving cross-modal feature alignment.

[0150] At the same time, the system also extracts the key semantic units involved in the data processing steps, feature matching process, and data flow path during the index retrieval process, and generates a visual decision tree based on these key units. The visual decision tree graphically displays the main logic and decision nodes in the retrieval process, allowing users to intuitively understand how the system finds relevant data based on their query request. For example, a decision tree might show how the system first screens a batch of candidate data based on text keywords, and then further filters and sorts them based on image features.

[0151] The system then generates natural language explanations and visual heatmaps based on the aligned semantic vectors and visual features. The natural language explanations use concise and clear text to explain the matching basis and relevance of the search results. For example, the system might explain that a result is highly semantically related to the query text and that a specific area in the image has similar visual features to the query image. The visual heatmap uses color coding to highlight the parts of the image most relevant to the query, helping users quickly locate key information.

[0152] Finally, the system integrates the generated visual decision tree, natural language explanation, and visual heat map to form a complete search result and send it to the user. This comprehensive search result not only provides accurate data matching, but also enhances the interpretability of the results through a variety of visualization methods, thereby improving the user experience. For example, when a user queries for images and text related to "beach sunset", the system not only returns matching images and articles, but also displays the search path through a decision tree, explains in text why these results are related to the query, and highlights key visual elements such as sunset and beach in the form of a heat map on the image. Through this method that combines multimodal feature extraction, feature alignment, and multiple visualization techniques, the system achieves accurate generation and interpretable display of server data retrieval results, providing users with a more intuitive and informative search experience.

[0153] According to the second aspect, the present application provides a server retrieval result generating device, such as Figure 8 As shown, the device includes:

[0154] The target data retrieval module 110 is used to, when receiving a user's query request, perform an index search on the source data of the target server using a preset hierarchical index to determine the target data that matches the query request. The hierarchical index is pre-built based on the source data.

[0155] The vector feature extraction module 120 is used to extract the semantic vector and visual features of the target data, and align the semantic vector and visual features according to a preset alignment rule.

[0156] The decision tree generation module 130 is used to extract key semantic units in the data processing steps, feature matching process and data flow path involved in index retrieval and generate a visual decision tree.

[0157] The heat map generation module 140 is used to generate a natural language explanation and a visualization heat map based on the aligned semantic vectors and visual features.

[0158] The search result generation module 150 is used to form the search results by combining the visual decision tree, natural language interpretation and visual heat map.

[0159] In some embodiments, before using a preset hierarchical index to index and retrieve the source data of the target server, the target data retrieval module 110 is also used to obtain source data from the target server, where the source data includes text data and image or video data. Based on the application scenario of the target server and the set regulatory rules, differential privacy technology is used to perform user privacy protection processing on the source data. Transfer learning technology is used to extract features from the source data after user privacy protection processing to assist in identifying and processing outliers and missing values. Crowdsourcing annotation technology is used to annotate and verify outliers and missing values ​​to obtain verified source data. Semantic vectors of text data and visual features of image or video data are extracted, and a hierarchical index is constructed based on the semantic vectors and visual features.

[0160] In some embodiments, the target data retrieval module 110 is further used to extract semantic vectors of text data using a preset BERT-like model, and to extract visual features of image or video data using a preset convolutional neural network-like model. The extracted text features and image features are aligned using a preset contrast loss and cosine similarity loss function, and the text features and image feature space are aligned using a clipboard-like architecture. Text features are feature vectors representing text semantic information in text data, and image features are feature vectors representing image or video content in image or video data. The image feature space is the multidimensional space in which the image features are located. A hierarchical inverted file index is constructed based on the aligned semantic vectors and visual features, which is used as a hierarchical index.

[0161] In some embodiments, when indexing and retrieving source data from a target server using a preset hierarchical index, the target data retrieval module 110 is further configured to monitor the distribution shift of the target data. When a distribution shift exceeding a preset threshold is detected, a model-agnostic meta-learning framework is used to pre-train a preset index retrieval model, and a two-layer architecture consisting of a student model and a teacher model is constructed as the architecture of the index retrieval model.

[0162] In some embodiments, when monitoring the distribution deviation of the target data, the target data retrieval module 110 is further used to maintain the current mean and covariance matrix statistics of the target data using a small batch sampling calculation method, maintain the current Wasserstein distance of the target data by setting a sliding window method, and set the preset threshold and deviation value for the current Wasserstein distance. When new target data appears, the mean and covariance matrix statistics of the target data are updated, and the Wasserstein distance of the new target data is calculated. If the distribution deviation of the new target data is within the set range, the preset threshold is reduced to the set first value. If the distribution deviation of the new target data exceeds the set range, the preset threshold is increased to the set second value.

[0163] In some embodiments, when constructing a two-layer architecture consisting of a student model and a teacher model, the target data retrieval module 110 is further configured to set the student model's dynamic distillation coefficient to decay exponentially over training time. The neural elastic weight consolidation and the ring buffer are configured based on a preset regularization coefficient for the neural elastic weight consolidation and a preset ring buffer capacity. The dynamic data drift detection and model update process of the index retrieval model is configured in a closed-loop feedback mode.

[0164] In some embodiments, the target data retrieval module 110 is further configured to compress the index retrieval model using dynamic channel pruning and / or quantization compression techniques. A performance test is performed on the compressed index retrieval model to obtain test results. The pruning rate and / or quantization bit number of the index retrieval model are adjusted based on the test results.

[0165] For specific definitions of the search result generating apparatus applicable to a server, please refer to the definitions of the search result generating method applicable to a server above and will not be repeated here. Each module in the above-mentioned search result generating apparatus applicable to a server can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in a computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0166] According to a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method for generating retrieval results of the server in any one of the above embodiments are implemented.

[0167] According to a fourth aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the computer device implements the steps of the server retrieval result generation method of any one of the above embodiments.

[0168] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 9 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data related to the generation of retrieval results of the server. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements any of the above-mentioned methods for generating retrieval results of the server.

[0169] In particular, any reference to memory, storage, database, or other media used in the various embodiments provided herein may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus (RamCUs), direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0170] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0171] The above embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

[0172] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

Claims

1. A method for generating search results by a server, characterized in that: include: In response to a user's query request, indexing and retrieving source data on a target server using a preset hierarchical index to determine target data that matches the query request; The hierarchical index is pre-built based on the source data; Extracting the semantic vector and visual features of the target data, and aligning the semantic vector and the visual features according to a preset alignment rule; Extract the key semantic units in the data processing steps, feature matching process and data flow path involved in index retrieval and generate a visual decision tree; generating a natural language explanation and a visual heat map based on the aligned semantic vectors and visual features; The visual decision tree, natural language explanation and visual heat map constitute the search results; When indexing and retrieving source data of a target server using a preset hierarchical index, the method further includes: monitoring the distribution deviation of the target data; when it is detected that the distribution deviation exceeds a preset threshold, pre-training a preset index retrieval model using a model-agnostic meta-learning framework, and constructing a two-layer architecture of a student model and a teacher model as the architecture of the index retrieval model; wherein the student model is used for actual reasoning and prediction tasks, the teacher model is used to generate knowledge to guide the learning and optimization of the student model, and the index retrieval model is a model for retrieving and processing server data; When monitoring the distribution deviation of the target data, the method also includes: maintaining the current mean and covariance matrix statistics of the target data using a small batch sampling calculation method, maintaining the current Wasserstein distance of the target data by setting a sliding window method, and setting the preset threshold and deviation value for the current Wasserstein distance; when new target data appears, updating the mean and covariance matrix statistics of the target data, and calculating the Wasserstein distance of the new target data; if the distribution deviation of the new target data is within a set range, reducing the preset threshold to a set first value; if the distribution deviation of the new target data exceeds the set range, enlarging the preset threshold to a set second value; wherein, the second value is greater than the first value.

2. The method according to claim 1, characterized in that Before performing index retrieval on the source data of the target server using the preset hierarchical index, the method further includes: Acquire the source data from the target server, wherein the source data includes text data and image or video data; Based on the application scenario of the target server and the set regulatory rules, the source data is processed using differential privacy technology to protect user privacy; Use transfer learning technology to extract features from source data after user privacy protection processing to assist in identifying and processing outliers and missing values; Using crowdsourcing annotation technology to annotate and verify the outliers and missing values, and obtain verified source data; Semantic vectors of text data and visual features of image or video data are extracted, and a hierarchical index is constructed based on the semantic vectors and the visual features.

3. The method according to claim 2, characterized in that The extracting of semantic vectors of text data and visual features of image or video data, and constructing a hierarchical index based on the semantic vectors and the visual features, includes: Use the preset BERT model to extract the semantic vector of text data, and use the preset convolutional neural network model to extract the visual features of image or video data; The extracted text features and image features are aligned using preset contrast loss and cosine similarity loss functions, and the text features and image feature spaces are aligned using a clipboard-like architecture; the text features are feature vectors representing text semantic information in the text data, the image features are feature vectors representing image or video content in the image or video data, and the image feature space is the multidimensional space where the image features are located; A hierarchical inverted file index is constructed according to the aligned semantic vectors and the visual features to serve as the hierarchical index.

4. The method according to claim 1, wherein When constructing a two-layer architecture of a student model and a teacher model, the method further includes: Setting the dynamic distillation coefficient of the student model to decay exponentially with training time; Setting the neural elastic weight consolidation and the ring buffer according to a preset regularization coefficient of the neural elastic weight consolidation and a preset capacity of the ring buffer; The dynamic data drift detection and model update process of the index retrieval model is set to a closed-loop feedback mode.

5. The method according to claim 1, wherein The method further comprises: Compressing the index retrieval model using dynamic channel pruning and / or quantization compression technology; Performing a performance test on the compressed index retrieval model to obtain a test result; The pruning rate and / or the number of quantization bits of the index retrieval model are adjusted according to the test results.

6. A retrieval result generating device of a server, characterized in that: include: The target data retrieval module is used to, when receiving a query request from a user, perform an index search on the source data of the target server using a preset hierarchical index to determine the target data that matches the query request; The hierarchical index is pre-built based on the source data; when the source data of the target server is indexed and retrieved using the preset hierarchical index, the distribution deviation of the target data is monitored; when it is detected that the distribution deviation exceeds the preset threshold, a model-agnostic meta-learning framework is used to pre-train the preset index retrieval model, and a two-layer architecture of a student model and a teacher model is constructed as the architecture of the index retrieval model; wherein the student model is used for actual reasoning and prediction tasks, the teacher model is used to generate knowledge to guide the learning and optimization of the student model, and the index retrieval model is a model for retrieving and processing server data; when monitoring the distribution deviation of the target data, a small batch is used to train the preset index retrieval model. The sampling calculation method maintains the current mean and covariance matrix statistics of the target data, maintains the current Wasserstein distance of the target data by setting a sliding window method, and sets the preset threshold and deviation value for the current Wasserstein distance; when new target data appears, the mean and covariance matrix statistics of the target data are updated, and the Wasserstein distance of the new target data is calculated; if the distribution deviation of the new target data is within a set range, the preset threshold is reduced to a set first value; if the distribution deviation of the new target data exceeds the set range, the preset threshold is increased to a set second value; wherein the second value is greater than the first value; A vector feature extraction module is used to extract the semantic vector and visual features of the target data, and align the semantic vector and the visual features according to a preset alignment rule; The decision tree generation module is used to extract the key semantic units in the data processing steps, feature matching process and data flow path involved in index retrieval and generate a visual decision tree; A heat map generation module, configured to generate a natural language explanation and a visualization heat map based on the aligned semantic vectors and visual features; The retrieval result generation module is used to form the retrieval results by combining the visual decision tree, natural language interpretation and visual heat map.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

Images