Vector database retrieval method for large vertical models in the marine field
By filtering and correcting similarity in the vector database of large-scale vertical models in the marine domain, and combining user behavior feedback and historical search experience, the search results were optimized, solving the problem of insufficient accuracy in the search of large-scale vertical models in the marine domain, and achieving more accurate search results.
Patent Information
- Application Number
- CN202611406325.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-10
- Publication Date
- 2026-11-10
AI Technical Summary
Existing vector database retrieval methods for large-scale vertical models in the marine field fail to effectively combine user behavior feedback with historical retrieval experience, resulting in insufficient retrieval accuracy, inability to identify users' true needs, and failure to explore the correlation between historical retrieval and current needs, leading to a waste of historical data value.
By filtering out matching historical retrieval vectors corresponding to the current retrieval vector from historical retrieval vectors, the information validity of each effective matching document vector recalled by each matching historical retrieval vector is determined, and corrections are made based on similarity and weight vectors. The matching document vectors recalled by the current retrieval vector are then filtered out, and the retrieval results are optimized by combining user behavior feedback and historical retrieval experience.
This improved the accuracy of search results in the marine field, ensuring that the results match users' actual needs and avoiding the placement of content with high semantic similarity but not relevant to user needs, thus enhancing search accuracy and user experience.
Smart Images

Figure CN123068616A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of semantic processing technology, specifically to a vector database retrieval method for large-scale vertical models in the marine field. Background Technology
[0002] With the rapid development of marine observation technology, satellite remote sensing monitoring, and marine scientific research systems, the marine field has accumulated a massive amount of multi-source heterogeneous data, covering various types of information such as hydrological time-series observations, remote sensing images, geological exploration, scientific research literature, and marine monitoring reports. This data serves as the core data support for intelligent marine analysis, marine environmental assessment, marine disaster prevention and mitigation, and marine scientific research.
[0003] The marine domain vertical large-scale model is a specialized large language model focusing on specific scenarios such as marine hydrology, marine meteorology, marine ecology, marine engineering, and marine disaster early warning. Compared with general-purpose large-scale models, it has stronger domain specialization, data adaptability, and scenario-based application capabilities. The core of vector database retrieval is to transform unstructured data (such as text and images) into high-dimensional vectors through an embedding model and store them. During retrieval, the query input is also transformed into a vector, and then its similarity to all vectors in the database (such as cosine similarity and Euclidean distance) is calculated in the vector space, returning the most similar results.
[0004] Existing problems: Traditional vector database retrieval for large-scale vertical models in the marine field relies solely on semantic similarity calculations, without incorporating user behavior feedback and historical retrieval experience; it only adapts to general semantic logic and does not optimize for the professional and complex semantics of marine terminology; it only sorts by vector matching degree, failing to identify users' true needs; and it fails to explore the correlation between historical searches and current needs, resulting in a waste of historical data value and insufficient retrieval accuracy and user experience. Summary of the Invention
[0005] This invention provides a vector database retrieval method for large vertical models in the marine field to solve existing problems.
[0006] The vector database retrieval method for large-scale vertical models in the marine field, as described in this invention, adopts the following technical solution: One embodiment of the present invention provides a vector database retrieval method for large vertical models in the marine field, the method comprising the following steps: Obtain the vector used for each effective retrieval recall for each document vector in the vector database of the large vertical model in the marine domain, as well as several effective matching document vectors recalled for each historical retrieval vector; Filter out several matching historical retrieval vectors corresponding to the current retrieval vector from the historical retrieval vectors; determine the weight vector of each document vector based on the similarity between different document vectors and the similarity between the usage vectors of each effective retrieval recall of different document vectors; determine the information validity relative to the current retrieval based on the usage vector and weight vector of each effective matching document vector recalled by each matching historical retrieval vector; Among all valid matching document vectors recalled by all historical retrieval vectors, the overall information validity relative to the current retrieval is determined based on the magnitude of the information validity of all valid matching document vectors belonging to the same document vector relative to the current retrieval. Combined with the number of all valid matching document vectors belonging to the same document vector, the reference validity relative to the current retrieval is determined. Based on the magnitude of the reference validity relative to the current retrieval, a similarity correction coefficient is determined for each document vector relative to the current retrieval; based on the similarity correction coefficient relative to the current retrieval, the similarity between the current retrieval vector and the document vector is corrected to obtain an effective similarity; based on the magnitude of the effective similarity, matching document vectors recalled by the current retrieval vector are selected.
[0007] Furthermore, the specific steps for selecting several matching historical retrieval vectors corresponding to the current retrieval vector from the historical retrieval vectors are as follows: Among the cosine similarities between the current search vector and all historical search vectors, all historical search vectors with a cosine similarity greater than a preset high similarity threshold are selected and recorded as matching historical search vectors.
[0008] Furthermore, the specific steps for determining the weight vector of each document vector are as follows: Obtain the mean vector of usage vectors for all valid retrieval recalls for each document vector, denoted as the average usage vector; Calculate the cosine similarity between any two document vectors, denoted as document content similarity. Then calculate the cosine similarity between the average usage vectors of the two document vectors, denoted as application similarity. Use the complement of the normalized values of the mean of document content similarity and application similarity as the clustering distance between the two document vectors. Perform clustering operation on all document vectors to obtain several document clusters. Obtain the mean vector of the average usage vector of all document vectors in each document cluster, and denote it as the standard usage vector; The weight vector for each document vector is determined based on the average usage vector, the standard usage vector, and the effective retrieval recall count for each document vector.
[0009] Furthermore, the specific steps for determining the weight vector of each document vector based on the average usage vector, the standard usage vector, and the effective retrieval recall count of each document vector are as follows: The weight vector of each document vector is obtained by weighting and summing the normalized value of the effective retrieval recall of each document vector as the weight of the average usage vector of each document vector, and the complement of the normalized value of the effective retrieval recall of each document vector as the weight of the standard usage vector of the document cluster to which each document vector belongs.
[0010] Furthermore, the specific steps for determining the validity of the information relative to the current retrieval are as follows: For each valid matching document vector recalled for each matching history retrieval vector, a weighted average is calculated based on all elements in the weight vector and all elements in the usage vector to obtain the information validity relative to the current retrieval.
[0011] Furthermore, the specific steps for determining the validity of the comprehensive information relative to the current retrieval are as follows: Among all the valid matching document vectors retrieved by all historical retrieval vectors, all valid matching document vectors belonging to the same document vector are combined into a set of valid matching document vectors. In each set of valid matching document vectors, the normalized value of the cosine similarity between the historical retrieval vector and the current retrieval vector corresponding to each valid matching document vector is denoted as retrieval consistency. The normalized value of the sum of all elements in the usage vector in the corresponding historical retrieval vector recall for each valid matching document vector is denoted as user attention. The average of retrieval consistency and user attention is denoted as the application weight of each valid matching document vector. This weighted average is used to calculate the information effectiveness of all valid matching document vectors relative to the current retrieval, thus obtaining the comprehensive information effectiveness of the same document vector corresponding to each set of valid matching document vectors relative to the current retrieval.
[0012] Furthermore, the specific steps for determining the reference validity relative to the current retrieval are as follows: The ratio of the number of valid matching document vectors in each type of valid matching document vector set to the number of historical matching retrieval vectors is recorded as the current frequency of occurrence of the same document vector in the same retrieval scenario corresponding to each type of valid matching document vector set. Based on preset information weights and preset frequency weights, the comprehensive information validity of the same document vector corresponding to each set of valid matching document vectors relative to the current retrieval and the current occurrence frequency of the same document vector in the same retrieval scenario are weighted and summed to obtain the reference validity of the same document vector corresponding to each set of valid matching document vectors relative to the current retrieval.
[0013] Furthermore, the specific steps for determining the similarity correction coefficient of each document vector relative to the current retrieval are as follows: The same document vector corresponding to each set of valid matching document vectors is denoted as the target document vector corresponding to the current retrieval vector; All document vectors other than the target document vector corresponding to the current retrieval vector are denoted as non-target document vectors corresponding to the current retrieval vector; Set the similarity correction coefficient of each non-target document vector corresponding to the current retrieval vector relative to the current retrieval to a preset baseline constant; The sum of the reference validity of each target document vector corresponding to the current retrieval vector relative to the current retrieval and the preset benchmark constant is denoted as the similarity correction coefficient of each target document vector corresponding to the current retrieval vector relative to the current retrieval.
[0014] Furthermore, the specific steps for obtaining effective similarity are as follows: The effective similarity is calculated by multiplying the normalized cosine similarity between the current search vector and each document vector by the similarity correction coefficient of each document vector relative to the current search vector.
[0015] Furthermore, the specific steps for filtering out the matching document vectors retrieved by the current retrieval vector are as follows: Among the effective similarities between the current retrieval vector and all document vectors, the top preset number of document vectors with the highest effective similarity are selected as the matching document vectors recalled by the current retrieval vector.
[0016] The beneficial effects of the technical solution of the present invention are: In this embodiment of the invention, several matching historical retrieval vectors corresponding to the current retrieval vector are selected from historical retrieval vectors. The validity of each effective matching document vector recalled by each matching historical retrieval vector relative to the information of the current retrieval is determined, thereby determining the reference validity relative to the current retrieval. This is used to determine the similarity correction coefficient of each document vector relative to the current retrieval, correcting the similarity between the current retrieval vector and the document vector to obtain effective similarity, and selecting the matching document vectors recalled by the current retrieval vector. This invention realizes the dynamic reuse of historical retrieval experience, and the retrieval effect is continuously optimized as users use it. By mining the correlation between historical retrieval and current retrieval, and reusing user behavior experience in the same scenario to dynamically optimize retrieval results, that is, by combining user behavior feedback and historical retrieval experience, it solves the problem of insufficient retrieval accuracy caused by the professionalism and complex semantics of marine terminology, and avoids the front placement of content with high semantic similarity but not user needs, thereby improving the retrieval accuracy in the marine field and ensuring that the retrieval results meet real needs. Attached Figure Description
[0017] Figure 1 This is a flowchart illustrating the steps of the vector database retrieval method for vertical large-scale models in the marine field, as described in this invention. Detailed Implementation
[0018] The following description, in conjunction with the accompanying drawings, details the specific scheme of the vector database retrieval method for large vertical models in the marine field provided by this invention.
[0019] Please see Figure 1 The diagram illustrates a flowchart of a vector database retrieval method for a large vertical model in the marine field, provided by an embodiment of the present invention. The method includes the following steps: Step S001: Obtain the vector used for each effective retrieval recall of each document vector in the vector database of the marine domain vertical large model, as well as several effective matching document vectors recalled by each historical retrieval vector.
[0020] First, construct a vector database for a large-scale vertical model of the ocean domain. The generally accepted procedure for constructing a vector database to support a large-scale vertical model of the ocean domain is as follows: First, multi-source heterogeneous data, including marine scientific research texts, time-series observations of temperature, salinity, and depth, and multi-modal remote sensing sonar data, are collected and uniformly loaded with marine-specific metadata such as sea area latitude and longitude, water depth, observation time, and subject classification. After format parsing, noise cleaning, and semantic segmentation by marine theme using a fixed block length and overlapping sliding window strategy, normalized high-dimensional vectors are generated by batch inference of text segments using a general embedding model or a vertical embedding model finely tuned to marine professional corpus. A storage unit containing unique identifiers, vectors, original text, and multi-dimensional scalar metadata is established. Then, the vector database set is initialized and the vector dimensions are matched. HNSW (Hierarchical Navigation Small World Map Index) or IVF (Inverted File Index) type vector retrieval index and keyword inverted index are built. Vector data is written in batches and the validity of vector dimensions, semantic recall, and spatiotemporal filtering is verified. Subsequent support includes incremental addition of new observations and scientific research data, cleaning of invalid data, optimization of retrieval performance, and continuous operation and maintenance iteration of hierarchical permission auditing, forming a marine vertical vector database that supports semantic vector retrieval and spatiotemporal scalar filtering.
[0021] Next, a user behavior data collection system will be built. Specifically: A front-end data collection module is embedded in the search interface to record in real time the various behavioral tag values of each user for each document vector retrieved by the search, including: (1) tags for staying to browse the full text. (Whether the full text of the document has been read is different from simply opening and quickly exiting), a mark of 1 indicates that the full text has been read, and a mark of 0 indicates that the full text has not been read; (2) Document scrolling progress mark (Indicating the degree of in-depth reading by the user), the page scrolling ratio is 1% to 100%. 1% means that the page does not scroll down after the document is opened, and only stays at the top of the document. 100% means that the page scrolls to the end of the document and all the content has been viewed; (3) Text selection, copy and excerpt markup (User-copied content, which belongs to strong positive intent), this behavior is marked as 1 if it exists, and marked as 0 if it does not exist; (4) Collection, saving or download mark (Perform local retention operation on the document), if this behavior exists, it is marked as 1, if this behavior does not exist, it is marked as 0; (5) Keyword search and location mark in the document (Search terms within the document), the presence of this behavior is marked as 1, and the absence of this behavior is marked as 0; (6) Cross-document jump tags (Users are redirected to the page by internal links in the document), this behavior is marked as 1 if it exists, and marked as 0 if it does not exist; (7) Repeated opening flag (Whether the document was repeatedly opened and viewed by the user in this search, with at least 2 opening and viewing times), this behavior is marked as 1 if it exists, and 0 if it does not exist; (8) Repeated search association marker (Whether the user's next search will continue to focus on the same document keywords) This behavior is marked as 1 if it exists, and 0 if it does not. Data is stored in association by search session (one user search constitutes one session). Each record contains a search vector, matching document vector, behavior tag, and behavior trigger time. A MySQL database is used to store historical searches, and a search vector index is built for rapid similarity calculation.
[0022] It should be noted that: for document vectors retrieved but not opened, and for which various behavioral tag values cannot be obtained, they are recorded as invalid matching document vectors in the retrieval recall, and the retrieval recall corresponding to such document vectors is considered invalid. For document vectors retrieved and opened, and for which various behavioral tag values can be obtained, they are recorded as valid matching document vectors in the retrieval recall, and the retrieval recall corresponding to such document vectors is considered valid.
[0023] Therefore, several document vectors can be obtained from the vector database of the large-scale vertical model in the marine domain, and the vector used for each effective retrieval recall of each document vector can be determined. And several historical search vectors of the user, and several valid matching document vectors recalled by each historical search vector.
[0024] It should be noted that: if the search content is the same (e.g., both are "tuna"), but the user ID or search timestamp is different, they belong to different search vectors. If the document vectors are the same, but the user ID or search recall timestamp is different, they belong to different valid matching document vectors. In this embodiment, the dimension, normalization rules, and encoding model of the search vectors are completely consistent with the generation rules of the document vectors, ensuring that they are in the same vector space for similarity matching in HNSW or IVF indexes.
[0025] Step S002: Select several matching historical retrieval vectors corresponding to the current retrieval vector from the historical retrieval vectors; determine the weight vector of each document vector based on the similarity between different document vectors and the similarity between the usage vectors of each effective retrieval recall of different document vectors; determine the information validity relative to the current retrieval based on the usage vector and weight vector of each effective matching document vector recalled by each matching historical retrieval vector.
[0026] It should be noted that text and data in the marine field are highly specialized and involve complex semantic relationships. Vector retrieval may fail to accurately understand domain terminology and concepts, leading to inaccurate search results. For example: (1) User query text: Observation results of vertical ventilation rate of subtropical modal water in the North Pacific during summer; Related original document text in the database: 2024 PN section hydrological survey, estimation of winter subsurface sinking, cross-density surface exchange flux and vertical mixing rate of mid-latitude subsurface modal water in the North Pacific; The similarity between the search vector and the document vector is moderate. This is because there are many variations in the expression of marine professional terms. The query term is "vertical ventilation rate", and the document uses "cross-density surface exchange flux, vertical mixing rate, sinking flux". The physical connotations of the two are completely consistent, but the literal words are partially the same. They cannot be aligned with the professional physical concepts of marine hydrology, resulting in the vector space being far apart, which leads to the related document content being placed at the back during retrieval.
[0027] (2) User query text: Nutrient transport flux of upwelling in the northern South China Sea shelf; Irrelevant original document text in the database: Observation of short-term fluctuations in surface nutrient concentration caused by vertical disturbances of water bodies due to tidal currents in the northern South China Sea shelf; The search vector and document vector have a high similarity because the two texts share a large number of words: northern South China Sea shelf, vertical, nutrients, water transport. However, their core scientific objectives are completely different: the former studies the long-term transport flux of upwelling (large-scale circulation process), while the latter only discusses the concentration oscillations caused by short-term disturbances of tidal currents (small-scale tidal process). They cannot distinguish the physical processes of different scales in ocean dynamics, resulting in false recall due to high similarity, which leads to irrelevant documents being prioritized in the search.
[0028] Therefore, in this embodiment, we first find historical search records that highly match the current search needs, analyze the user's behavior in the historical search records that highly match the current search needs, and determine the document content that the user really needs, so as to solve the problem of insufficient search accuracy caused by the professionalism and complex semantics of marine terminology.
[0029] Get the user's current search vector.
[0030] Select several matching historical search vectors corresponding to the current search vector from all historical search vectors.
[0031] It should be noted that the current search vector and all matching historical search vectors should belong to the same search scenario. If the number of matching historical search vectors is less than 3, it means that the current search scenario itself has few historical search records, and the cosine similarity between the current search vector and the document vector is directly used as the effective similarity.
[0032] It should be further noted that for any valid matching document vector recalled from any historical retrieval vector, the larger the value of each behavior tag in the corresponding usage vector, the more likely the valid matching document vector is to be the relevant document that the user is currently searching for. However, different documents carry different information values, and the effective behavior patterns of users are significantly different: (1) Reference materials: Users only quickly browse and consult them, and rarely copy, excerpt, or repeatedly open them. In this case, the value of such behaviors should be increased. and Weighting, suppressing and (2) Data summary documents: Users only extract fragments of content and do not need to read the whole text. The weight of data summary documents should be increased. The weight is reduced. and The weight of (3) In-depth research literature: will be accompanied by multiple openings, internal searches, cross-document jumps, and secondary searches, and the weight of these literature needs to be increased accordingly. , , as well as The weighting of user interest is crucial. Applying a uniform, fixed weight to all behavioral tags will smooth out the differences in business attributes across documents, ultimately distorting the calculation of user interest. Configuring an independent weight coefficient for each behavior based on document type allows for precise quantification of the user's true level of interest in different documents.
[0033] Therefore, the weight vector of each document vector is determined based on the similarity between different document vectors in the vector database of the vertical large model of the marine domain, and the similarity between the vectors used in each effective retrieval recall of different document vectors.
[0034] Then, based on the usage vector and weight vector of each valid matching document vector recalled for each matching history retrieval vector, the validity of the information relative to the current retrieval is determined.
[0035] Step S003: Among all valid matching document vectors recalled by all historical retrieval vectors, determine the overall information validity relative to the current retrieval based on the magnitude of the information validity of all valid matching document vectors belonging to the same document vector relative to the current retrieval. Combine this with the number of all valid matching document vectors belonging to the same document vector to determine the reference validity relative to the current retrieval.
[0036] It should be noted that the same document vector in the vector database of the large vertical model in the marine domain is often recalled multiple times in the same retrieval scenario. Therefore, it is necessary to further analyze the effectiveness of the comprehensive information of the same document vector recalled multiple times by matching historical retrieval vectors relative to the current retrieval.
[0037] Among all valid matching document vectors recalled from all historical retrieval vectors, the overall information validity relative to the current retrieval is determined based on the relative validity of all valid matching document vectors belonging to the same document vector relative to the information validity of the current retrieval.
[0038] The more valid matching document vectors belonging to the same document vector, the higher the general value of the document in the current retrieval scenario.
[0039] Therefore, by combining the number of all valid matching document vectors belonging to the same document vector, the reference validity relative to the current retrieval is determined.
[0040] Step S004: Determine the similarity correction coefficient of each document vector relative to the current retrieval based on the magnitude of the reference validity relative to the current retrieval; correct the similarity between the current retrieval vector and the document vector based on the similarity correction coefficient relative to the current retrieval to obtain effective similarity; filter out the matching document vectors recalled by the current retrieval vector based on the magnitude of the effective similarity.
[0041] It should be noted that: if the current search query is "Observation results of vertical ventilation rate of subtropical modal water in the North Pacific during summer", among all the matching historical search queries corresponding to the current search query, the relevant document in the database, "2024 PN section hydrological survey, estimation of winter subsurface deposition, cross-density surface exchange flux and vertical mixing rate of mid-latitude subsurface modal water in the North Pacific", is often used effectively by users multiple times, i.e., its reference effectiveness is relatively high. Therefore, a higher similarity correction coefficient is assigned to this relevant document in the database to improve the moderate similarity between the search vector and the original document vector. Conversely, if the current search query is "Nutrient transport flux of upwelling in the northern shelf of the South China Sea", among all the matching historical search queries corresponding to the current search query, the irrelevant document in the database, "Observation of short-term fluctuations in surface nutrient concentration caused by tidal currents in the northern shelf of the South China Sea", is often ignored by users, i.e., its reference effectiveness is relatively low. Therefore, a lower similarity correction coefficient is assigned to this irrelevant document in the database to reduce the high similarity between the search vector and the original document vector. This can solve the problem of insufficient search accuracy caused by the professionalism and complex semantics of marine terminology.
[0042] It should be further noted that because these irrelevant documents lack effective user behavior feedback, their calculated reference validity is relatively small, and the corresponding similarity correction coefficient is also small, which will reduce the similarity. Compared with those truly relevant documents that contain a large amount of historical behavior and have amplified similarity, these irrelevant documents will lag behind in the subsequent effective similarity descending order, thus eliminating their original high basic matching advantage brought about by literal overlap.
[0043] Therefore, based on the calculated reference validity relative to the current retrieval, the similarity correction coefficient of each document vector in the vector database of the marine domain vertical large model relative to the current retrieval is determined.
[0044] Therefore, based on the similarity correction coefficient of each document vector in the vector database of the vertical large model of the marine domain relative to the current retrieval, the similarity between the current retrieval vector and the document vector is corrected to obtain the effective similarity between the current retrieval vector and the document vector.
[0045] Finally, based on the magnitude of the effective similarity between the current retrieval vector and all document vectors in the vector database of the large-scale vertical model in the marine domain, the matching document vectors recalled by the current retrieval vector are selected.
[0046] This enables the dynamic reuse of historical search experience, and the search results are continuously optimized as users use the system. By mining the correlation between historical searches and current searches, the search results are dynamically optimized by reusing user behavior experience in the same scenario.
[0047] Preferably, in some possible implementations of the embodiments of the present invention, the cosine similarity between the current search vector and each historical search vector is calculated. Among the cosine similarities between the current search vector and all historical search vectors, all historical search vectors with a cosine similarity greater than a preset high similarity threshold are obtained and recorded as matching historical search vectors.
[0048] It should be noted that the calculation of cosine similarity is a well-known technique, and the specific method will not be introduced here. The value of cosine similarity ranges from -1 to 1. When the cosine similarity is 1, the two vectors are completely in the same direction and have a high degree of semantic consistency.
[0049] Preferably, in some possible implementations of the embodiments of the present invention, the mean vector of all valid retrieval recalls of each document vector in the vector database of the vertical large model of the marine domain is obtained, and denoted as the average usage vector.
[0050] It should be noted that the average usage vector reflects the user behavior of the document. The larger the value of each element in the average usage vector, the more frequently the corresponding user behavior occurred during user retrieval, indicating that the user behavior corresponding to that element better matches the application attributes of the document and should be assigned a larger weight. In this embodiment, document vectors without effective retrieval are not subjected to subsequent clustering analysis and weight vector acquisition. The vector database of the large-scale vertical model in the marine domain has undergone extensive experimental retrieval to ensure that it contains a large number of document vectors with effective retrieval.
[0051] For any two document vectors in the vector database of the large-scale vertical model in the marine domain, calculate the cosine similarity between the two document vectors, denoted as document content similarity. Then calculate the cosine similarity between the average usage vectors of the two document vectors, denoted as application similarity. The mean of document content similarity and application similarity is then calculated. normalized value complement , which is the clustering distance between any two document vectors, is used to perform clustering operations on all document vectors using a hierarchical clustering algorithm to obtain several document clusters.
[0052] Hierarchical clustering is a well-known technique, and its specific method will not be described here. In this embodiment, the upper limit threshold for the merging distance of cluster trees is set to 0.3, and this will be used as an example for description. Since the cosine similarity value ranges from -1 to 1, Between -1 and 1, therefore As The normalized value.
[0053] Obtain the mean vector of the average usage vector of all document vectors in each document cluster, denoted as the standard usage vector.
[0054] It should be noted that the standard usage vector reflects the standard user application performance of the same type of document.
[0055] Preferably, in some possible implementations of the embodiments of the present invention, the normalized value of the effective retrieval recall of each document vector is used as the weight of the average usage vector of each document vector, and the complement of the normalized value of the effective retrieval recall of each document vector is used as the weight of the standard usage vector of the document cluster to which each document vector belongs. The weights are then summed to obtain the weight vector of each document vector. Specifically: No. Weight vector of document vectors The calculation formula is:
[0056] In the formula, For the first Effective retrieval recall for each document vector For the first The maximum effective retrieval recall among all document vectors in the document cluster to which a given document vector belongs. for The normalized value, for complement, For the first Average usage vector of document vectors For the first The standard vector used to identify the document cluster to which a document vector belongs.
[0057] It should be noted that: Can represent the first Personalized weight vectors for each document vector. Can represent the first The standard weight vector of the type to which each document vector belongs. A larger value indicates that the document is frequently and effectively recalled. The weight vector is mainly dominated by the personalized weight vector, prioritizing personalized metrics formed by the document's own historical behavior. The smaller the value, the less likely the document is to be effectively retrieved. The personalized weight has very low reliability, and the weight vector is mainly dominated by the general standard weight vector of the cluster, supplementing the data by borrowing the general behavioral characteristics of documents of the same category.
[0058] Preferably, in some possible implementations of the embodiments of the present invention, for the first Matching historical retrieval vectors The recall 1 valid matching document vector ,according to weight vector All elements in, for Using vectors We take a weighted average of all elements to obtain The validity of the information retrieved. The specific calculation formula is as follows:
[0059] In the formula, for The Middle Each element value for The Middle Each element value. Due to the effective matching of the document vector, the document scroll progress marker is used in the vector. The value ranges from 1% to 100%, meaning it is not 0. Therefore, the second element in the weight vector is not 0, hence the denominator... Not zero.
[0060] Preferably, in some possible implementations of the embodiments of the present invention, among all valid matching document vectors recalled by all matching historical retrieval vectors, all valid matching document vectors belonging to the same document vector are combined to form a set of valid matching document vectors, thus obtaining several sets of valid matching document vectors.
[0061] In the Among all valid matching document vectors recalled by the matching history retrieval vector, calculate the first... The valid matching document vector at the th th The sum of all elements in the vector used in the historical retrieval vector recall. normalized value This is recorded as user attention. .
[0062] in, In the first The maximum sum of all elements in the vector used in the matching history retrieval vector recall is used to determine the maximum value among all valid matching document vectors. The larger the value, the more attention users pay to this document in this search, meaning the more useful the document is in this search.
[0063] In the In the set of document vectors that are validly matched by class, the first one will be... The cosine similarity between the historical retrieval vectors and the current retrieval vector corresponding to each valid matching document vector. normalized value This is denoted as retrieval consistency. ,Will With the User attention corresponding to each valid matching document vector The mean of, as the first The application weights of each valid matching document vector The weighted average of all valid matching document vectors relative to the information validity of the current retrieval is used to obtain the first... The effectiveness of the same document vector corresponding to the set of document vectors that effectively matches a class relative to the overall information retrieved in the current search. The specific calculation formula is as follows:
[0064] In the formula, For the first The number of valid matching document vectors in the set of valid matching document vectors for a class. For the first The class effectively matches the first document vector in the document vector set. The applied weights correspond to the valid matching document vectors. For the first The class effectively matches the first document vector in the document vector set. The validity of a valid matching document vector relative to the information retrieved.
[0065] It should be noted that: since the value of cosine similarity ranges from -1 to 1, therefore... As The normalized value. The cosine similarity between the matched historical retrieval vector and the current retrieval vector is greater than 0.8. Due to the use of document scrolling progress markers in the vector, the effective matching document vectors are obtained. The value ranges from 1% to 100%, meaning it is not zero. Therefore, the user attention level is greater than 0, hence the denominator is... Not zero. If The larger it is, the more likely it is to be the first The greater the reference value of the retrieval behavior corresponding to each valid matching document vector, and the more focused the user is on this document in this retrieval behavior, the more it needs to be assigned... Higher weights are applied, and a weighted average is then performed to obtain the overall information effectiveness of the same document vector relative to the current retrieval.
[0066] Preferably, in some possible implementations of the embodiments of the present invention, the first... Number of valid matching document vectors in the set of valid matching document vectors for each class Number of matching historical retrieval vectors ratio , denoted as the The frequency of occurrence of the same document vector in the same retrieval scenario corresponding to the set of document vectors that effectively match the class .
[0067] Preset information weights The preset frequency weight is 0.7. It is 0.3. Let's take this as an example.
[0068] Based on preset information weights With preset frequency weights ,right and Perform a weighted summation to obtain the first... The reference validity of the same document vector corresponding to the set of valid matching document vectors of a class relative to the current retrieval. The specific calculation formula is as follows:
[0069] It should be noted that: through this process... and By performing a weighted summation, we can highlight the value of personalized needs reflected in user behavior while also taking into account the general value of the document in similar scenarios.
[0070] Preferably, in some possible implementations of the embodiments of the present invention, the same document vector corresponding to each set of valid matching document vectors is denoted as the target document vector corresponding to the current retrieval vector.
[0071] In the vector database of the large vertical model in the marine domain, all document vectors other than the target document vector corresponding to the current retrieval vector are denoted as non-target document vectors corresponding to the current retrieval vector.
[0072] The default baseline constant is 1, and this will be used as an example for explanation.
[0073] In the vector database of the vertical large model in the marine domain, the similarity correction coefficient of each non-target document vector corresponding to the current retrieval vector relative to the current retrieval is set as a preset benchmark constant 1; the sum of the reference validity of each target document vector corresponding to the current retrieval vector relative to the current retrieval and the preset benchmark constant 1 is denoted as the similarity correction coefficient of each target document vector corresponding to the current retrieval vector relative to the current retrieval.
[0074] Thus, the similarity correction coefficient of each document vector in the vector database of the large vertical model in the marine domain is obtained relative to the current retrieval.
[0075] Preferably, in some possible implementations of the embodiments of the present invention, the cosine similarity between the current retrieval vector and each document vector in the vector database of the large-scale vertical model in the marine domain is obtained. normalized value ,Will The product of the similarity correction coefficient for each document vector with respect to the current retrieval is denoted as the effective similarity.
[0076] Since the cosine similarity value ranges from -1 to 1, therefore... As The normalized value.
[0077] Preferably, in some possible implementations of the embodiments of the present invention, the preset recall quantity is 10, which will be described as an example. The effective similarity between the current retrieval vector and all document vectors in the vector database of the large-scale vertical model in the marine domain is obtained, and the top 10 document vectors with the highest effective similarity are taken as the matching document vectors recalled by the current retrieval vector.
[0078] This invention is now complete.
[0079] In summary, in this embodiment of the invention, several matching historical retrieval vectors corresponding to the current retrieval vector are selected from historical retrieval vectors. The validity of each effective matching document vector recalled by each matching historical retrieval vector relative to the current retrieval is determined, thereby determining the reference validity relative to the current retrieval. This reference validity is used to determine the similarity correction coefficient of each document vector relative to the current retrieval, correcting the similarity between the current retrieval vector and the document vector to obtain effective similarity, and selecting the matching document vectors recalled by the current retrieval vector. This invention can improve retrieval accuracy.
Claims
1. A vector database retrieval method applied to large-scale vertical models in the marine field, characterized in that, The method includes the following steps: Obtain the vector used for each effective retrieval recall for each document vector in the vector database of the large vertical model in the marine domain, as well as several effective matching document vectors recalled for each historical retrieval vector; Filter out several matching historical retrieval vectors corresponding to the current retrieval vector from the historical retrieval vectors; determine the weight vector of each document vector based on the similarity between different document vectors and the similarity between the usage vectors of each effective retrieval recall of different document vectors; determine the information validity relative to the current retrieval based on the usage vector and weight vector of each effective matching document vector recalled by each matching historical retrieval vector; Among all valid matching document vectors recalled by all historical retrieval vectors, the overall information validity relative to the current retrieval is determined based on the magnitude of the information validity of all valid matching document vectors belonging to the same document vector relative to the current retrieval. Combined with the number of all valid matching document vectors belonging to the same document vector, the reference validity relative to the current retrieval is determined. Based on the magnitude of the reference validity relative to the current retrieval, a similarity correction coefficient is determined for each document vector relative to the current retrieval; based on the similarity correction coefficient relative to the current retrieval, the similarity between the current retrieval vector and the document vector is corrected to obtain an effective similarity; based on the magnitude of the effective similarity, matching document vectors recalled by the current retrieval vector are selected.
2. The vector database retrieval method for large-scale vertical models in the marine field according to claim 1, characterized in that, The specific steps for selecting several matching historical search vectors corresponding to the current search vector from historical search vectors are as follows: Among the cosine similarities between the current search vector and all historical search vectors, all historical search vectors with a cosine similarity greater than a preset high similarity threshold are selected and recorded as matching historical search vectors.
3. The vector database retrieval method for large-scale vertical models in the marine field according to claim 1, characterized in that, The specific steps for determining the weight vector for each document vector are as follows: Obtain the mean vector of usage vectors for all valid retrieval recalls for each document vector, denoted as the average usage vector; Calculate the cosine similarity between any two document vectors, denoted as document content similarity. Then calculate the cosine similarity between the average usage vectors of the two document vectors, denoted as application similarity. Use the complement of the normalized values of the mean of document content similarity and application similarity as the clustering distance between the two document vectors. Perform clustering operation on all document vectors to obtain several document clusters. Obtain the mean vector of the average usage vector of all document vectors in each document cluster, and denote it as the standard usage vector; The weight vector for each document vector is determined based on the average usage vector, the standard usage vector, and the effective retrieval recall count for each document vector.
4. The vector database retrieval method for large-scale vertical models in the marine field according to claim 3, characterized in that, The specific steps for determining the weight vector of each document vector based on the average usage vector, the standard usage vector, and the effective retrieval recall count of each document vector are as follows: The weight vector of each document vector is obtained by weighting and summing the normalized value of the effective retrieval recall of each document vector as the weight of the average usage vector of each document vector, and the complement of the normalized value of the effective retrieval recall of each document vector as the weight of the standard usage vector of the document cluster to which each document vector belongs.
5. The vector database retrieval method for large-scale vertical models in the marine field according to claim 1, characterized in that, The specific steps involved in determining the validity of the information relative to the current retrieval are as follows: For each valid matching document vector recalled for each matching history retrieval vector, a weighted average is calculated based on all elements in the weight vector and all elements in the usage vector to obtain the information validity relative to the current retrieval.
6. The vector database retrieval method for large-scale vertical models in the marine field according to claim 1, characterized in that, The specific steps involved in determining the validity of the comprehensive information relative to the current retrieval are as follows: Among all the valid matching document vectors retrieved by all historical retrieval vectors, all valid matching document vectors belonging to the same document vector are combined into a set of valid matching document vectors. In each set of valid matching document vectors, the normalized value of the cosine similarity between the historical retrieval vector and the current retrieval vector corresponding to each valid matching document vector is denoted as retrieval consistency. The normalized value of the sum of all elements in the usage vector in the corresponding historical retrieval vector recall for each valid matching document vector is denoted as user attention. The average of retrieval consistency and user attention is denoted as the application weight of each valid matching document vector. This weighted average is used to calculate the information effectiveness of all valid matching document vectors relative to the current retrieval, thus obtaining the comprehensive information effectiveness of the same document vector corresponding to each set of valid matching document vectors relative to the current retrieval.
7. The vector database retrieval method for large-scale vertical models in the marine field according to claim 6, characterized in that, The specific steps for determining the reference validity relative to the current retrieval are as follows: The ratio of the number of valid matching document vectors in each type of valid matching document vector set to the number of historical matching retrieval vectors is recorded as the current frequency of occurrence of the same document vector in the same retrieval scenario corresponding to each type of valid matching document vector set. Based on preset information weights and preset frequency weights, the comprehensive information validity of the same document vector corresponding to each set of valid matching document vectors relative to the current retrieval and the current occurrence frequency of the same document vector in the same retrieval scenario are weighted and summed to obtain the reference validity of the same document vector corresponding to each set of valid matching document vectors relative to the current retrieval.
8. The vector database retrieval method for large-scale vertical models in the marine field according to claim 7, characterized in that, The specific steps for determining the similarity correction coefficient of each document vector relative to the current retrieval are as follows: The same document vector corresponding to each set of valid matching document vectors is denoted as the target document vector corresponding to the current retrieval vector; All document vectors other than the target document vector corresponding to the current retrieval vector are denoted as non-target document vectors corresponding to the current retrieval vector; Set the similarity correction coefficient of each non-target document vector corresponding to the current retrieval vector relative to the current retrieval to a preset baseline constant; The sum of the reference validity of each target document vector corresponding to the current retrieval vector relative to the current retrieval and the preset benchmark constant is denoted as the similarity correction coefficient of each target document vector corresponding to the current retrieval vector relative to the current retrieval.
9. The vector database retrieval method for large-scale vertical models in the marine field according to claim 1, characterized in that, The specific steps involved in obtaining effective similarity are as follows: The effective similarity is calculated by multiplying the normalized cosine similarity between the current search vector and each document vector by the similarity correction coefficient of each document vector relative to the current search vector.
10. The vector database retrieval method for large-scale vertical models in the marine field according to claim 1, characterized in that, The specific steps for filtering out the matching document vectors retrieved by the current search vector are as follows: Among the effective similarities between the current retrieval vector and all document vectors, the top preset number of document vectors with the highest effective similarity are selected as the matching document vectors recalled by the current retrieval vector.
