Vector retrieval method and device for multiple service scenes, storage medium and program product

Abstract

The invention discloses a vector retrieval method and device for multiple service scenes, a storage medium and a program product, and relates to the technical field of big data. The method comprises the following steps: receiving a target retrieval request, determining a modal type of the target retrieval request, and determining a target retrieval vector matched with the target retrieval request based on the modal type; determining a retrieval range in a predetermined target material library, and determining a candidate material identifier set; wherein the target material library comprises materials of a plurality of business scenes and feature vectors corresponding to the materials; respectively determining the similarity between the target retrieval vector and the feature vector of each candidate material in the candidate material identification set; and based on the similarity calculation result, determining a target candidate material identifier, and determining a material content corresponding to the target candidate material identifier as a retrieval result of the target retrieval request. According to the scheme, storage resource waste and vectorization calculation overhead can be reduced, cross-scene joint retrieval is achieved, and the retrieval breadth and flexibility are improved.

Description

Technical Field

[0001] This invention relates to the field of big data technology, and in particular to a vector retrieval method, device, storage medium, and program product for multiple business scenarios. Background Technology

[0002] With the rapid development of artificial intelligence and big data technologies, cross-modal retrieval systems have been widely used in e-commerce, content platforms, intelligent customer service, and other fields. These systems typically transform unstructured data such as text, images, and audio into high-dimensional feature vectors through deep learning models and store them in a vectorized database, enabling intelligent retrieval functions such as "searching for images by text" and "searching for videos by images."

[0003] However, in practical applications, enterprises often have multiple independent business scenarios (e.g., product databases, news material databases, user-generated content databases, etc.), and different scenarios have different requirements for retrieval performance, data visibility, and update frequency. Currently, data isolation is achieved by building a separate vector database for each business scenario. While this approach facilitates access control, it leads to the repeated storage and computation of the same content across multiple databases, resulting in wasted storage resources and increased vectorization computation overhead. Furthermore, it cannot support cross-scenario joint retrieval, limiting the breadth and flexibility of retrieval.

[0004] How to reduce storage resource waste and vectorized computing overhead, achieve cross-scenario joint retrieval, and improve the breadth and flexibility of retrieval are key research issues in the industry. Summary of the Invention

[0005] This invention provides a vector retrieval method, device, storage medium, and program product for multiple business scenarios, in order to reduce storage resource waste and vectorization computation overhead, realize cross-scenario joint retrieval, and improve the breadth and flexibility of retrieval.

[0006] According to one aspect of the present invention, a vector retrieval method for multiple service scenarios is provided, executed by a data center service, the method comprising:

[0007] Receive a target retrieval request, determine the modality type of the target retrieval request, and determine the target retrieval vector that matches the target retrieval request based on the modality type;

[0008] The search scope is determined in a pre-defined target material library, and a candidate material identifier set is determined based on the search scope; wherein, the target material library includes materials for multiple business scenarios, as well as feature vectors corresponding to each material;

[0009] Determine the similarity between the target retrieval vector and the feature vectors of each candidate material in the candidate material identifier set;

[0010] Based on the similarity calculation results, the target candidate material identifier is determined, and the material content corresponding to the target candidate material identifier is determined as the retrieval result of the target retrieval request.

[0011] Optionally, determining the modality type of the target retrieval request and determining the target retrieval vector matching the target retrieval request based on the modality type includes:

[0012] Modality recognition is performed on the target retrieval request to determine the modality type of the target retrieval request;

[0013] Based on the modality type, a target encoding model matching the target retrieval request is determined, and the target retrieval request is processed based on the target encoding model to obtain a target retrieval vector matching the target retrieval request;

[0014] The modal type includes at least one of the following: text, image, video, and audio.

[0015] Optionally, the target material library is determined based on the following steps:

[0016] Obtain material sets for different business scenarios; wherein, the material sets include at least one of the following data: text data, image data, video data, and audio data;

[0017] Based on the content information of each material in the material set, a scene identifier and a unique material identifier corresponding to each material are determined respectively;

[0018] Based on the scene identifier of the target material, query the preset scene table to obtain the priority information of the business scene corresponding to the scene identifier;

[0019] Based on the target material identifier, the target fragment in the first database cluster corresponding to the target material is determined;

[0020] Determine whether there exists a reference material identifier in the target segment that is identical to the target material identifier of the target material;

[0021] If it exists, the target material will be identified as a duplicate material;

[0022] If it does not exist, then based on the target material and the target material identifier, a fixed-dimensional feature vector is generated, and the target material identifier and the feature vector are combined to form a target key-value pair;

[0023] The target key-value pairs are stored in the vector storage area, and the attribute information of the target material is stored in the metadata area; the attribute information includes: format, size, or type;

[0024] Write the target material identifier to the target segment. If the number of elements in the target segment reaches the upper limit, write the target material identifier to the next segment that is not yet full.

[0025] In response to the completion instruction for writing the material identifiers of all materials, the target material library is obtained.

[0026] Optionally, before obtaining the target media library in response to the completion instruction for writing media identifiers of all media, the method further includes:

[0027] In response to the call command of the target interface, the scenario table is queried according to the scenario identifier to obtain the priority information of the business scenario;

[0028] If the priority is high, the target segment is determined, and the target segment is located in the first database cluster. The material identifier is written into the integer set of the segment.

[0029] If the priority is low, the association between scene identifiers and material identifiers is written to the cold data area of ​​the database layer buffer pool and the association between scene identifiers and material identifiers is persisted to the scene identifier-material identifier list in the database.

[0030] In response to the start command of the data center service, if a buffer pool image file is detected in the preset path, the scene identifier-material identifier list saved in the image file is loaded into the buffer pool hot data area; if it does not exist, the associated data of high-priority and low-priority scenes marked as hot are read from the database and loaded into the first database cluster and the buffer pool hot data area respectively.

[0031] In response to the shutdown command of the data center service, the list of scene identifiers and material identifiers currently stored in the buffer pool hot data area is serialized and written to a preset path to generate or update the buffer pool image file.

[0032] Optionally, determining the search scope in a predetermined target material library and determining a candidate material identifier set based on the search scope includes:

[0033] Determine the target retrieval scenario corresponding to the target retrieval request, and determine the retrieval scope based on the target retrieval scenario; wherein, the retrieval scope includes high-priority scenarios or low-priority scenarios;

[0034] For each scene identifier within the search scope, a set of candidate material identifiers is determined based on the priority marking information and hot marking information of the scene identifier.

[0035] Optionally, for each scene identifier within the search scope, a candidate material identifier set is determined based on the priority marking information and hot marking information of the scene identifier, including:

[0036] If the priority corresponding to the scene identifier is 1, then the target segment is determined, the target segment is located in the first database cluster, and the integer set structure in the target segment and subsequent non-empty segments is traversed to extract the first material identifier stored therein;

[0037] If the priority is 0 and the hot tag is 1, then read the scene identifier-material identifier list stored in the form of a continuous integer array from the hot data area of ​​the database layer buffer pool to obtain the second material identifier set;

[0038] If the priority is 0 and the hot tag is 0, then the scenario is excluded from the current candidate set construction, or delayed until subsequent on-demand access.

[0039] The first material identifier and the second material identifier are merged, and global deduplication is performed to obtain the candidate material identifier set.

[0040] Optionally, determining the target candidate material identifier based on the similarity calculation result, and determining the material content corresponding to the target candidate material identifier as the retrieval result of the target retrieval request, includes:

[0041] The candidate material identifiers are sorted according to the similarity calculation results, and the sorting results are weighted and adjusted in combination with scene priority and hot tags. A preset number of material identifiers with the highest ranking are selected as target candidate material identifiers.

[0042] Based on the target candidate material identifier, the corresponding original material content is obtained from the metadata area of ​​the vectorized main library, and the original material content is returned as the retrieval result of this retrieval request.

[0043] According to another aspect of the present invention, a vector retrieval device for multiple service scenarios is provided, deployed in a data center service, the device comprising:

[0044] The receiving module is used to receive the target retrieval request, determine the modality type of the target retrieval request, and determine the target retrieval vector that matches the target retrieval request based on the modality type.

[0045] The retrieval scope determination module is used to determine the retrieval scope in a pre-determined target material library and determine a candidate material identifier set based on the retrieval scope; wherein, the target material library includes materials for multiple business scenarios and feature vectors corresponding to each material;

[0046] The similarity determination module is used to determine the similarity between the target retrieval vector and the feature vectors of each candidate material in the candidate material identifier set.

[0047] The retrieval result determination module is used to determine the target candidate material identifier based on the similarity calculation results, and to determine the material content corresponding to the target candidate material identifier as the retrieval result of the target retrieval request.

[0048] According to another aspect of the present invention, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute a vector retrieval method for multi-service scenarios according to any embodiment of the present invention.

[0049] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement the vector retrieval method for multiple service scenarios according to any embodiment of the present invention.

[0050] According to another aspect of the present invention, a computer program product is provided, including a computer program that, when executed by a processor, implements a vector retrieval method for multiple service scenarios according to any embodiment of the present invention.

[0051] The solution of this invention receives a target retrieval request, determines the modality type of the target retrieval request, and determines a target retrieval vector matching the target retrieval request based on the modality type; this helps in determining the subsequent retrieval results; it determines the retrieval scope in a pre-determined target material library and determines a candidate material identifier set based on the retrieval scope; wherein, the target material library includes materials from multiple business scenarios and feature vectors corresponding to each material; by determining the candidate material identifier set and determining the final retrieval result from the candidate material identifier set, the retrieval scope is narrowed, and the retrieval speed and accuracy are improved; the similarity between the target retrieval vector and the feature vectors of each candidate material in the candidate material identifier set is determined respectively; based on the similarity calculation result, the target candidate material identifier is determined, and the material content corresponding to the target candidate material identifier is determined as the retrieval result of the target retrieval request, which can reduce storage resource waste and vectorization calculation overhead, realize cross-scenario joint retrieval, and improve the breadth and flexibility of retrieval.

[0052] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart of a vector retrieval method for multiple service scenarios provided according to Embodiment 1 of the present invention;

[0055] Figure 2 This is a flowchart illustrating the construction process of a target material library according to Embodiment 2 of the present invention;

[0056] Figure 3 This is a flowchart of a vector retrieval method for multiple service scenarios provided according to Embodiment 2 of the present invention;

[0057] Figure 4 This is a schematic diagram of the structure of a vector retrieval device for multiple service scenarios provided in Embodiment 3 of the present invention;

[0058] Figure 5 This is a schematic diagram of the structure of an electronic device that implements the vector retrieval method for multiple service scenarios according to embodiments of the present invention. Detailed Implementation

[0059] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0060] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0061] Example 1

[0062] Figure 1 This is a flowchart of a vector retrieval method for multiple business scenarios according to Embodiment 1 of the present invention. This embodiment is applicable to situations where search results can be quickly determined in a target material library containing multiple business vendors. This method can be executed by a data center service built into electronic devices such as computers and servers. Specifically, refer to... Figure 1 The method specifically includes the following:

[0063] Step 110: Receive the target retrieval request, determine the modality type of the target retrieval request, and determine the target retrieval vector that matches the target retrieval request based on the modality type.

[0064] The target retrieval request can be a data input containing a query intent submitted by a user or business system. It can be voice data input, text data input, video data input, or image data input, or it can be a combination of different modal data input, such as a combination of image data and text data. This embodiment does not limit it.

[0065] Optionally, in this embodiment, after receiving a target retrieval request, the data center service built into the electronic device can further determine the modality type of the target retrieval request, i.e., the modality of the input query information. In this embodiment, the modality type of the target retrieval request may include one or more of text, images, videos, and audio, and is not limited thereto. Furthermore, a matching target retrieval vector can be determined based on the modality type of the target retrieval request.

[0066] Optionally, in this embodiment, determining the modality type of the target retrieval request and determining the target retrieval vector matching the target retrieval request based on the modality type may include: performing modality recognition on the target retrieval request to determine the modality type of the target retrieval request; determining the target encoding model matching the target retrieval request based on the modality type; and processing the target retrieval request based on the target encoding model to obtain the target retrieval vector matching the target retrieval request.

[0067] In an optional implementation of this embodiment, after receiving the target retrieval request, modality recognition can be further performed on the target retrieval request to determine its modality type. For example, the modality type of the target retrieval request can be text, image, video, audio, text and image, or text and video, etc., and this embodiment does not limit it.

[0068] After determining the modality type of the target retrieval request, a target encoding model matching the target retrieval request can be further determined based on the modality type, and the target retrieval request can be processed based on the target encoding model to obtain the target retrieval vector matching the target retrieval request.

[0069] In an optional implementation of this embodiment, after identifying the modality type of the target retrieval request, a target encoding model matching the modality type can be further determined based on a preset modality-model mapping relationship. Further, the query content (i.e., the target query request) can be input into the target encoding model, an initial feature vector can be generated through forward inference, and L2 normalization can be performed on it to obtain a target retrieval vector matching the target retrieval request; wherein the target encoding models corresponding to modality types such as text, image, audio, and video can be Sentence-BERT, ViT, Wav2Vec, and VideoCLIP, respectively.

[0070] Step 120: Determine the search scope in the pre-determined target material library, and determine the candidate material identifier set based on the search scope.

[0071] The target content library includes content from multiple business scenarios, as well as feature vectors corresponding to each content. It should be noted that the target content library involved in this embodiment is not a physically independent database, but rather a content pool that supports retrieval from different business scenarios. It integrates unstructured content from multiple business scenarios and achieves efficient and intelligent content organization and retrieval through mechanisms such as vectorization, deduplication, and association management.

[0072] Optionally, in this embodiment, the search range can be determined after the target search vector is obtained, or the search range can be determined directly based on the target search request after the target search request is received. This embodiment does not limit either approach.

[0073] In this embodiment, the search scope can be either high-priority scenarios or low-priority scenarios.

[0074] Optionally, in this embodiment, determining the search scope in a predetermined target material library and determining a candidate material identifier set based on the search scope may include: determining the target search scenario corresponding to the target search request, and determining the search scope based on the target search scenario; wherein, the search scope includes high-priority scenarios or low-priority scenarios; for each scenario identifier within the search scope, determining a candidate material identifier set based on the priority marking information and hot marking information of the scenario identifier.

[0075] In an optional implementation of this embodiment, after receiving the target retrieval request, the request can be further parsed to determine the scene identifier of the target retrieval request, and the target retrieval scene corresponding to the target retrieval request can be determined based on the scene identifier. For example, the target retrieval scene could be searching for similar short videos on a content platform, searching for brand materials for enterprise digital asset management, searching for intelligent customer service solutions, or searching for the same products on an e-commerce platform. Furthermore, the retrieval scope can be determined based on the target retrieval scene; for example, if the target scene belongs to a high-priority scene, then the retrieval scope is the high-priority scene.

[0076] In another optional implementation of this embodiment, if the target retrieval request does not specify a scene identifier, i.e. the target retrieval scene cannot be determined, then a set of candidate scenes can be determined according to a preset default retrieval strategy; wherein, the default retrieval strategy includes: only including high-priority scenes with priority 1, or further including low-priority scenes with hot-marked 1.

[0077] After determining the search scope, we can further identify each scene identifier within the search scope, and determine the candidate material identifier set based on the priority marking information and hot marking information of each scene identifier.

[0078] In one optional implementation of this embodiment, for each scene identifier within the retrieval scope, a candidate material identifier set is determined based on the priority marking information and hot marking information of the scene identifier. This may include: if the priority corresponding to the scene identifier is 1, then a target segment is determined, and the target segment is located in the first database cluster. The integer set structure in the target segment and subsequent non-empty segments is traversed to extract the first material identifier stored therein; if the priority is 0 and the hot marking is 1, then the scene identifier-material identifier list stored in the form of a continuous integer array is read from the hot data area of ​​the database layer buffer pool to obtain the second material identifier set; if the priority is 0 and the hot marking is 0, then the scene is excluded from the current candidate set construction, or delayed until subsequent on-demand access; the first material identifier and the second material identifier are merged, and global deduplication is performed to obtain the candidate material identifier set.

[0079] Optionally, in this embodiment, for high-priority scenarios with priority 1, the first database cluster, such as the target shard in the Redis cluster, can be located based on the scenario identifier (ID) hash. The integer set (intset) in the shard and its subsequent non-empty shards is then traversed to extract the first material identifier stored therein, ensuring fast loading of high-frequency key data. For low-priority hot scenarios with priority 0 but hot flag 1, the scenario-material mapping list (referred to as the scenario identifier-material identifier list in this embodiment) stored in a continuous array form is directly read from the hot data area of ​​the database layer buffer pool to obtain the second material identifier set. For scenarios that are both low-priority and non-hot (priority=0 and hot_flag=0), they are not included in the candidate set in the initial stage to avoid occupying memory. They are only loaded from the database on demand during subsequent actual access, realizing a lazy loading mechanism. Finally, the material identifiers from Redis and the hot data area are merged and global deduplication is performed to form a candidate material identifier set that matches the target retrieval request, ensuring accurate and efficient retrieval.

[0080] In one example of this embodiment, a user on a news aggregation platform searches for "Shenzhou-19 launch site video," requiring the construction of a candidate material set corresponding to the search content. Scenario A is the "Current Affairs News Channel" (priority=1). The system locates the target shard in the Redis cluster based on its scenario ID hash value and iterates through the integer set structure in that shard and subsequent non-empty shards, extracting all associated news video material IDs as the first material identifier. Scenario B is the "Tech Enthusiast Community" (priority=0 but hot_flag=1). Due to its recent high popularity, its material ID list has been pre-loaded into the Buffer-Pool hot data area, and the system directly reads this continuous array to obtain the second material identifier set. Scenario C, the "Historical Archives" (priority=0 and hot_flag=0), is not included in this candidate set construction due to its low access frequency; it is only loaded on demand when the user clicks "View More Historical Content" later. Finally, the system merges the first and second material identifiers and removes duplicates to form a candidate material set. This ensures that users can quickly retrieve the latest video content about the Shenzhou launch from mainstream media and popular communities.

[0081] Step 130: Determine the similarity between the target retrieval vector and the feature vectors of each candidate material in the candidate material identifier set.

[0082] Optionally, in this embodiment, after determining the candidate material identifier set, the candidate material corresponding to each candidate material identifier in the candidate material identifier set can be determined, and the feature vector of each candidate material can be determined (which can be obtained from the target material library). Further, the cosine similarity between the target retrieval vector and the feature vector of each candidate material can be calculated.

[0083] In the specific implementation, the vectorized main library can be accessed based on the candidate material identifier, and the pre-stored feature vectors can be read one by one. Cosine similarity is used to measure the semantic similarity between the target retrieval vector and each candidate feature vector. The similarity value can be in the range of [-1, 1], with the value closer to 1 indicating greater similarity.

[0084] Step 140: Based on the similarity calculation results, determine the target candidate material identifier, and determine the material content corresponding to the target candidate material identifier as the retrieval result of the target retrieval request.

[0085] Optionally, in this embodiment, after calculating the similarity results, the similarity results can be sorted from high to low, and a number of top-ranked material identifiers (e.g., 20, 50, or 60) can be selected as target candidate material identifiers; these are then sorted from high to low according to their similarity results. After sorting, the Top-K best-matching material identifiers are finally selected as the target candidate set. Subsequently, based on these target candidate material identifiers, the corresponding material content (e.g., images, videos, text, etc.) can be obtained from the metadata area of ​​the vectorized master library or the original storage, and encapsulated into structured results according to the similarity results, and returned to the user as the final search result for this target search request.

[0086] In one optional implementation of this embodiment, determining the target candidate material identifier based on the similarity calculation result and determining the material content corresponding to the target candidate material identifier as the retrieval result of the target retrieval request may include: sorting the candidate material identifiers according to the similarity calculation result, and adjusting the sorting result by combining scene priority and hot tag, selecting a preset number of material identifiers with high sorting as target candidate material identifiers; obtaining the corresponding original material content from the metadata area of ​​the vectorized master library according to the target candidate material identifier, and returning the original material content as the retrieval result of this retrieval request.

[0087] In practice, candidate material identifiers can be sorted in descending order based on the similarity results between the target retrieval vector and the feature vectors of each candidate material to initially determine the matching priority. Subsequently, a weighted adjustment is made in combination with business dimension information (for hot content with hot_flag=1, the weight is appropriately increased based on the original similarity score to move it to the front of the list, ensuring that current hot events or high-attention content are presented first).

[0088] After weighted sorting, a preset number of material identifiers (e.g., Top-20) can be selected as the final target candidate material identifiers. Finally, the corresponding original material content (e.g., news videos, images, titles, body text, etc.) is extracted from the metadata area of ​​the vectorized master library using these identifiers, encapsulated in order, and returned to the user to achieve intelligent search results that are both semantically relevant and conform to business weights.

[0089] In a specific example of this embodiment, a user searches for "highlights of the closing ceremony of a sporting event," and a batch of candidate news video materials is calculated and sorted by similarity. Among them, videos from the "Sports Channel" (priority=1) and the "Hot Topics Page" (hot_flag=1) receive a weighted boost on top of their original similarity scores, improving their ranking from 5th and 8th to 1st and 2nd. Further, the top 10 ranked materials are selected as the target candidate set, and the corresponding high-definition video clips, titles, and publication times are retrieved from the metadata area, and this content is returned to the user in sequence.

[0090] The solution in this embodiment receives a target retrieval request, determines the modality type of the target retrieval request, and determines a target retrieval vector matching the target retrieval request based on the modality type; this helps in determining the subsequent retrieval results; it determines the retrieval scope in a pre-determined target material library, and determines a set of candidate material identifiers based on the retrieval scope; wherein, the target material library includes materials from multiple business scenarios, and feature vectors corresponding to each material; by determining the set of candidate material identifiers and determining the final retrieval results from the set of candidate material identifiers, the retrieval scope is narrowed, and the retrieval speed and accuracy are improved; the similarity between the target retrieval vector and the feature vectors of each candidate material in the set of candidate material identifiers is determined respectively; based on the similarity calculation results, the target candidate material identifier is determined, and the material content corresponding to the target candidate material identifier is determined as the retrieval result of the target retrieval request, which can reduce storage resource waste and vectorization calculation overhead, realize cross-scenario joint retrieval, and improve the breadth and flexibility of retrieval.

[0091] Example 2

[0092] Figure 2 This is a flowchart illustrating the construction process of a target material library according to Embodiment 2 of the present invention. This embodiment is a further refinement of the above technical solution, and the technical solution in this embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 2 As shown, the construction process of the target material library includes:

[0093] Step 210: Obtain material sets for different business scenarios.

[0094] The material set includes at least one of the following types of data: text data, image data, video data, and audio data.

[0095] In this embodiment, the business scenario may include breaking news reporting, in-depth investigative reporting, or multimedia converged reporting, etc., and is not limited thereto in this embodiment.

[0096] Optionally, in this embodiment, material sets for different business scenarios can be obtained through different databases; for example, material sets for different business scenarios can be obtained through news databases or news media websites, thereby forming the material sets involved in this embodiment.

[0097] Step 220: Based on the content information of each material in the material set, determine the scene identifier and unique material identifier corresponding to each material.

[0098] Optionally, in this embodiment, after obtaining the material set, the content information of each material in the material set can be obtained separately, and the scene identifier and material identifier corresponding to each material can be determined based on the content information. In this embodiment, these are referred to as unique material identifiers. It is understood that the scene identifiers of material A and material B may be the same or different; however, their material identifiers must be different. This allows each material to be identified in the target material library, which is beneficial for subsequent material deduplication.

[0099] Step 230: Based on the scene identifier of the target material, query the preset scene table to obtain the priority information of the business scene corresponding to the scene identifier.

[0100] The target material can be any material in the material set, and this embodiment does not limit it. In this embodiment, the preset scene table contains all scene identifiers and priority information corresponding to each scene identifier.

[0101] In an optional implementation of this embodiment, after determining the scene identifier of the target material, the priority information of the business scene corresponding to the scene identifier of the target sauerkraut can be queried from a preset scene table. The priority information can be high priority or low priority, and this embodiment does not limit it.

[0102] Step 240: Determine the target fragment in the first database cluster corresponding to the target material based on the target material identifier.

[0103] The first database cluster can be a Redis cluster or other distributed databases; this embodiment does not limit it.

[0104] In one optional implementation of this embodiment, the target material identifier can be used as input to calculate the modulo operation result with the preset number of fragments; for example, target material identifier % 1000 yields the target fragment number; and the corresponding physical node in the first database cluster consisting of 1000 physical fragments is located based on the number.

[0105] For example, if the target material's material identifier is 500123, by calculating 500123 % 1000 = 123, it is determined that it should belong to the 123rd physical shard of the Redis cluster.

[0106] Step 250: Determine whether there is a reference material identifier in the target segment that is the same as the target material identifier of the target material; if it exists, the target material is identified as a duplicate material; if it does not exist, generate a fixed-dimensional feature vector based on the target material and the target material identifier, and combine the target material identifier and the feature vector to form a target key-value pair.

[0107] Optionally, in this embodiment, after determining that the target segment has been obtained, deduplication verification or data writing operations can be further performed in the integer set (intset) of the target segment to ensure that the same material identifier is always routed to the same segment.

[0108] In the specific implementation, it is possible to query whether the same reference material identifier already exists in the target segment; if it exists, the material is determined to be a duplicate material and the processing is terminated; if it does not exist, a fixed-dimensional feature vector is generated based on the material content, and the feature vector and the material identifier are combined to form a target key-value pair for subsequent vectorized retrieval and similarity matching, thereby realizing an efficient processing mode of deduplication first and then vectorization, reducing redundant computational overhead.

[0109] In one example of this embodiment, if the target material is an image and the material identifier of the target material is 300456, calculating 300456 % 1000 = 456, locating the 456th shard in the Redis cluster, querying its deduplication set dedup_set:456, and finding that the ID does not exist, it is determined to be a new material; subsequently, a 512-dimensional feature vector is extracted using the ResNet-50 model, and this vector is combined with the material identifier to form a key-value pair {key: "material:300456", value: [0.87, -0.32, ..., 0.45]}, which is written to the vector database. At the same time, the material identifier is added to the deduplication set to ensure that subsequent requests can identify duplicates.

[0110] The solution in this embodiment adopts a single-vector library and a hierarchical storage architecture based on business relationships. It achieves data deduplication through globally unique material identifiers, completely solving the resource overload and data silo problems caused by the "one business, one library" approach in traditional solutions.

[0111] Step 260: Store the target key-value pairs in the vector storage area and store the attribute information of the target material in the metadata area.

[0112] The attribute information includes: format, size, or type.

[0113] In the target material library constructed in this embodiment, a hierarchical storage architecture can be adopted. The feature vectors of the target key-value pairs are written to the vector storage area, and the attribute information of the target materials is written to the metadata area. The vector storage area stores fixed-dimensional feature vectors generated from the material content and establishes a vector index to support efficient semantic similarity retrieval. The metadata area stores the structured attribute information of the materials, including business fields such as title, source, publication time, scene identifier, and material type. It is understood that both areas use the target material identifier as a unique association key to achieve unified data management and collaborative access.

[0114] Step 270: Write the target material identifier to the target segment. If the number of elements in the target segment reaches the upper limit, write the target material identifier to the next segment that is not yet full.

[0115] Optionally, in this embodiment, when writing the target material identifier into the first database cluster, the main target fragment number is first calculated based on the target material identifier % 1000, and the corresponding physical fragment is located; then, it is checked whether the current number of elements in the integer set in the fragment has reached a preset upper limit (e.g., 100, 200, or 500, which is not limited in this embodiment); if it has not reached the upper limit, the target material identifier is directly written into the target fragment; if the upper limit has been reached, the subsequent fragments are traversed in ascending order of fragment number, and the target material identifier is written into the first non-empty fragment that is not full.

[0116] The advantage of this setup is that it ensures the continuity and reliability of write operations, enabling elastic storage based on the overflow chain mechanism.

[0117] Furthermore, in response to the call command of the target interface, the system can query the scene table based on the scene identifier to obtain the priority information of the business scene; if the priority is high, the target shard is determined and located in the first database cluster, and the material identifier is written into the integer set of the shard; if the priority is low, the association between the scene identifier and the material identifier is written into the cold data area of ​​the database layer buffer pool and the association between the scene identifier and the material identifier is persisted to the scene identifier-material identifier list in the database; in response to the start command of the data center service, if a buffer pool image file is detected in the preset path, the scene identifier-material identifier list saved in the image file is loaded into the buffer pool hot data area; if it does not exist, the association data of high-priority and low-priority scenes marked as hot are read from the database and loaded into the first database cluster and the buffer pool hot data area respectively; in response to the shutdown command of the data center service, the scene identifier-material identifier list currently stored in the buffer pool hot data area is serialized and written into the preset path, and the buffer pool image file is generated or updated.

[0118] The target interface refers to a network service interface that can be used to receive requests to associate and bind target materials with business scenarios. Its calling instructions carry scenario identifiers and material identifiers and are used to trigger the index building process.

[0119] Optionally, in this embodiment, after receiving the call instruction of the target interface, the scenario table in the database can be queried according to the scenario identifier carried in the request to obtain the priority attribute (such as high / low) and hot tag (whether it is a hot scenario) corresponding to the business scenario. If the scenario priority is high, it indicates that it is extremely sensitive to retrieval latency, and a high-performance write path can be executed immediately: for example, based on the material identifier to be written, the target shard number is obtained by calculating material identifier%1000, and the shard with the corresponding number is located in the first database cluster (such as a Redis cluster) consisting of 1000 physical nodes. The material identifier is written into the integer set (intset) maintained by the shard. Redis's memory storage and efficient set operation capabilities are used to support subsequent millisecond-level candidate set generation and deduplication query. If the scene priority is low, it indicates that the access frequency is low or a certain delay is acceptable. The system adopts a lower-cost write strategy: write the association between scene identifier and material identifier to the cold data area of ​​the database layer buffer pool, and asynchronously or in batches persist the mapping relationship to the scene identifier-material identifier list of the underlying database, avoiding frequent access to high-performance storage resources and realizing reasonable allocation of resources.

[0120] To further improve system startup efficiency and service availability, after receiving the startup command for the data center service, the system can first check if a buffer pool image file exists in a preset path (such as / data / buffer_pool_image.bin). If it exists, the serialized scene identifier-material identifier mapping list in the file is directly loaded into the hot data area of ​​the buffer pool to quickly restore access capabilities for high-frequency scenes and avoid performance jitter caused by cold starts. If the image file does not exist, the system reverts to database loading mode, reads all high-priority scenes and associated data of low-priority scenes marked as hot from the underlying database, and loads them into the first database cluster (for high-priority scenes) and the hot data area of ​​the buffer pool (for hot low-priority scenes) respectively to ensure access performance of critical paths.

[0121] Furthermore, upon receiving a shutdown command for the data center service, the system can proactively serialize the currently valid and frequently accessed scene identifier-material identifier mapping list in the buffer pool's hot data area and write it to a preset path, generating or updating the buffer pool image file to provide rapid recovery capabilities for the next service startup. This process can significantly improve the stability, response speed, and operational efficiency of the retrieval system in high-concurrency, multi-scenario environments.

[0122] In one example of this embodiment, when a news recommendation interface call with scene identifier 1001 is received, a query of the scene table reveals that its priority is high. Therefore, for the new material identifier 500123, the algorithm 500123 % 1000 = 123 is executed, locating the 123rd shard in Redis and writing it to the intset set of that shard. For a special report with scene identifier 2005 and low priority, the association relationship of its material identifier 600789 is written to the cold data area of ​​the buffer pool and persisted to the database. Upon service restart, if the image file / data / buffer_pool_image.bin exists, the hot mapping list within it is directly loaded into the hot data area; otherwise, scene data with high priority or hot marker 1 is queried from the database and loaded into Redis and the buffer pool respectively. Before service shutdown, the latest scene identifier-material identifier list in the hot data area is serialized and saved to this path to ensure rapid recovery of high-frequency access capabilities upon next startup.

[0123] The solution in this embodiment stores high-priority business scenario relationships in Redis shards, while low-priority scenario relationships are managed through a Buffer-Pool array-based memory area. It also supports DB configuration preloading and mirroring mechanisms: when the service starts, it prioritizes loading the image file or the DB-specified hot scenario data, and when it shuts down, it automatically generates a Buffer-Pool image to ensure that high retrieval performance can still be maintained after restarting.

[0124] Step 280: In response to the completion instruction for writing the material identifiers of all materials, the target material library is obtained.

[0125] Optionally, in this embodiment, once all the material identifiers have been written, the target material library involved in this embodiment can be obtained.

[0126] In one optional implementation of this embodiment, before creating the target material library, a globally unique material storage pool can be created, which includes a vector storage area and a metadata area, and the material identifier is a 64-bit integer (globally unique).

[0127] Data Center Service: Redis Cluster Configuration: Divide into 1000 physical shards (numbered 0~999), each shard is forced to use an integer set (intset), and the number of elements is limited by set-max-intset-entries=1000, which is used to store the global deduplication set and the association relationship of high priority scenarios.

[0128] Database layer Buffer-Pool configuration:

[0129] As a memory acceleration layer for the database, it divides the capacity into a 60% hot data area and a 40% cold data area, both storing scene identifier-material identifier lists as contiguous integer arrays, supporting the following:

[0130] Configure the preload list (preload_scene_flags): Specify the low-priority scenes I (such as the department's core material library) that need to be preloaded at startup;

[0131] Image file path (buffer_pool_image_path): Used for the buffer-pool image when the storage service is shut down.

[0132] Database layer table structure:

[0133] Scene table: includes scene identifier, scene name, priority (1 for high, 0 for low), and hot marker (1 for hot scene, manually set by the business side).

[0134] Relationship table: includes scene identifier, material identifier, and validity status. The composite primary key is scene identifier + material identifier.

[0135] Business scenario preloading and caching configuration:

[0136] After the data center service is started, data preheating is completed according to a fixed procedure:

[0137] 1. Load images first:

[0138] 1.1 Check if the image file exists in buffer_pool_image_path (generated when the service was last shut down). If it exists, directly load the "scene identifier - material identifier list" in the image into the buffer-pool hot data area (skip the DB query); if it does not exist, execute the DB preloading process.

[0139] 2. DB preloading (without image):

[0140] 2.1 High-priority scenarios (priority=1): Read the scenario identifier and associated material identifier from the DB, and write them to the Redis shard according to the scenario identifier modulo 1000 (if the number of elements is 1000, it will be rolled to the next shard).

[0141] 2.2. DB-marked hot scenes (low-priority scenes with hot marker = 1): Read the associated material identifiers from the DB and write them into the buffer-pool hot data area as an integer array.

[0142] 2.3 Other low-priority scenarios: Not loaded at the moment; read from the DB and written to the buffer-pool cold data area on the first access.

[0143] 3. Image generation when the service is shut down:

[0144] 3.1 When the service receives a shutdown command, it automatically serializes the "scene identifier-material identifier list" of the buffer-pool hot data area to buffer_pool_image_path, generates an image file (overwriting the old image), and ensures that it can be reused directly on the next startup.

[0145] Before the business side pushes materials to the vectorized master database, a fixed deduplication process is performed by the data center service:

[0146] 1. Generate a 64-bit integer material identifier based on the original material data;

[0147] 2. Calculate the material identifier modulo 1000, locate the target fragment of the global deduplication set, and traverse the fragment integer set to check for duplicates;

[0148] 3. If there are no duplicates, push the data to the vectorized main library. After the vector storage is completed, write the material identifier to the global deduplication set (if the segment is full, continue to the next segment) and synchronize it to the DB relationship table.

[0149] Maintaining the association between business scenarios and material identifiers is mainly achieved through the following steps:

[0150] The business side calls the add_relation(scene_identifier, material_identifier) ​​interface:

[0151] High-priority scenarios (priority=1): Redis shards are located based on scenario identifier %1000, and if they are full, they are postponed to the next shard and updated synchronously to the database;

[0152] Low-priority scenario (priority=0): Write to the buffer-pool cold data area and update to the DB synchronously (the DB's own caching mechanism ensures the efficiency of subsequent access).

[0153] The solution in this embodiment updates Redis and DB in real time for high-priority scenarios and updates Buffer-Pool and DB synchronously for low-priority scenarios. Combined with a sharding modulo expansion mechanism and an asynchronous persistence strategy, it supports ultra-large-scale business scenarios while ensuring consistency of relationships and operational performance, and reducing business adaptation and maintenance costs.

[0154] In this embodiment, after the target material library is created, upon receiving a search request initiated by the business side, the search can be performed through the following steps:

[0155] 1. Obtaining the collection of material identifiers:

[0156] 1.1 High-priority scenarios: Traverse the Redis shards corresponding to scenario identifier %1000 and subsequent non-empty shards to collect material identifiers;

[0157] 1.2 Low-priority scenarios: If it is a hot-marking scenario, the buffer-pool hot data area array is read directly; if it is a non-hot-marking scenario, it is read from the DB and then written to the buffer-pool cold data area.

[0158] 1.3 Merge identifiers and remove duplicates to generate a search list.

[0159] 2. Vector Matching: The vectorized main library calculates the cosine similarity between the retrieved vector and the material vectors in the list, and returns the Top 100 results.

[0160] In this embodiment, during business-side retrieval, the data center service quickly retrieves a set of material identifiers from Redis shards or Buffer-Pool based on scenario priority, and then passes it to the vector library for vector matching. This avoids multiple queries and result distortion in cross-database retrieval. At the same time, batch processing of identifier sets significantly improves cross-scenario retrieval efficiency and unlocks the value of data synergy.

[0161] The process of deleting a relationship may include: calling the delete_relation(scene_identifier, material_identifier) ​​interface:

[0162] 1. High-priority scenario: Traversing the corresponding Redis shard deletion flags and synchronously updating the database is ineffective.

[0163] 2. Low-priority scenario: Removing the identifier from the buffer-pool array renders asynchronous DB updates ineffective.

[0164] To better understand the vector retrieval method for multiple service scenarios involved in the embodiments of the present invention, Figure 3 This is a flowchart of a vector retrieval method for multiple business scenarios according to Embodiment 2 of the present invention. It mainly includes a business service layer, a data center service scheduling layer, and a vectorized master library. By constructing a multi-layered caching and vectorized matching material processing and retrieval system, high-performance, scalable, and intelligent content management and retrieval capabilities are achieved. Specifically, Redis is used for rapid response to high-priority material ID sets, Buffer-Pool enables asynchronous preloading of low-priority tasks and cold data storage, a relational database ensures persistent storage of material metadata and relationships, and the vectorized master library supports efficient semantic-level retrieval based on content understanding. Overall, this architecture not only supports millisecond-level access to hot materials, optimized resource allocation and task scheduling, but also enables intelligent matching and management of large-scale materials. It is suitable for various application scenarios requiring high concurrency, low latency, and semantic understanding, such as short video recommendation, advertising material deduplication, and content review. The final result is the creation of a high-efficiency content platform with functions such as rapid response, intelligent matching, task priority scheduling, hot and cold data separation, and data consistency assurance.

[0165] Example 3

[0166] Figure 4 This is a schematic diagram of a vector retrieval device for multiple service scenarios according to Embodiment 3 of the present invention. This device can be deployed in data center services. Figure 4 As shown, the device includes: a receiving module 410, a search range determination module 420, a similarity determination module 430, and a search result determination module 440.

[0167] The receiving module 410 is used to receive a target retrieval request, determine the modality type of the target retrieval request, and determine a target retrieval vector that matches the target retrieval request based on the modality type.

[0168] The retrieval scope determination module 420 is used to determine the retrieval scope in a pre-determined target material library and determine a candidate material identifier set based on the retrieval scope; wherein, the target material library includes materials for multiple business scenarios and feature vectors corresponding to each material;

[0169] The similarity determination module 430 is used to determine the similarity between the target retrieval vector and the feature vectors of each candidate material in the candidate material identifier set;

[0170] The retrieval result determination module 440 is used to determine the target candidate material identifier based on the similarity calculation result, and determine the material content corresponding to the target candidate material identifier as the retrieval result of the target retrieval request.

[0171] In this embodiment, the receiving module receives a target retrieval request, determines the modality type of the target retrieval request, and determines a target retrieval vector matching the target retrieval request based on the modality type. The retrieval scope determination module determines the retrieval scope within a pre-defined target material library and determines a set of candidate material identifiers based on the retrieval scope. The target material library includes materials from multiple business scenarios and feature vectors corresponding to each material. The similarity determination module determines the similarity between the target retrieval vector and the feature vectors of each candidate material in the candidate material identifier set. The retrieval result determination module determines the target candidate material identifier based on the similarity calculation result and identifies the material content corresponding to the target candidate material identifier as the retrieval result of the target retrieval request. This approach reduces storage resource waste and vectorization computation overhead, enables cross-scenario joint retrieval, and improves the breadth and flexibility of the retrieval.

[0172] In an optional implementation of this embodiment, the receiving module 410 is specifically used to perform modal recognition on the target retrieval request and determine the modal type of the target retrieval request;

[0173] Based on the modality type, a target encoding model matching the target retrieval request is determined, and the target retrieval request is processed based on the target encoding model to obtain a target retrieval vector matching the target retrieval request;

[0174] The modal type includes at least one of the following: text, image, video, and audio.

[0175] In an optional implementation of this embodiment, the vector retrieval device for multiple business scenarios further includes a target material library construction module, used for:

[0176] Obtain material sets for different business scenarios; wherein the material sets include at least one of the following data: text data, image data, video data, and audio data;

[0177] Based on the content information of each material in the material set, the scene identifier and unique material identifier corresponding to each material are determined respectively;

[0178] Based on the scene identifier of the target material, query the preset scene table to obtain the priority information of the business scene corresponding to the scene identifier;

[0179] Based on the target material identifier, the target fragment in the first database cluster corresponding to the target material is determined;

[0180] Determine if a reference material identifier exists in the target segment that is identical to the target material identifier of the target material;

[0181] If it exists, the target material will be identified as a duplicate material;

[0182] If it does not exist, a fixed-dimensional feature vector is generated based on the target material and the target material identifier, and the target material identifier and the feature vector are combined to form a target key-value pair;

[0183] The target key-value pairs are stored in the vector storage area, and the attribute information of the target material is stored in the metadata area; the attribute information includes: format, size or type;

[0184] Write the target material identifier to the target segment. If the number of elements in the target segment reaches the upper limit, write the target material identifier to the next segment that is not full.

[0185] In response to the completion command for writing all material identifiers, the target material library is obtained.

[0186] In an optional implementation of this embodiment, the target material library construction module is further configured to:

[0187] In response to the call command of the target interface, query the scenario table based on the scenario identifier to obtain the priority information of the business scenario;

[0188] If the priority is high, the target segment is determined, and the target segment is located in the first database cluster. The material identifier is written into the integer set of the segment.

[0189] If the priority is low, the association between scene identifiers and material identifiers will be written to the cold data area of ​​the database layer buffer pool and the association between scene identifiers and material identifiers will be persisted to the scene identifier-material identifier list in the database.

[0190] In response to the start command of the data center service, if a buffer pool image file is detected in the preset path, the scene identifier-material identifier list saved in the image file is loaded into the buffer pool hot data area; if it does not exist, the associated data of high-priority and low-priority scenes marked as hot are read from the database and loaded into the first database cluster and the buffer pool hot data area respectively.

[0191] In response to the shutdown command of the data center service, the list of scene identifiers and material identifiers currently stored in the buffer pool hot data area is serialized and written to a preset path, and a buffer pool image file is generated or updated.

[0192] In an optional implementation of this embodiment, the retrieval range determination module 420 is specifically used to determine the target retrieval scenario corresponding to the target retrieval request, and determine the retrieval range based on the target retrieval scenario; wherein, the retrieval range includes high-priority scenarios or low-priority scenarios;

[0193] For each scene identifier within the search scope, a set of candidate material identifiers is determined based on the priority marking information and hot marking information of the scene identifier.

[0194] In an optional implementation of this embodiment, the retrieval range determination module 420 is further specifically used to determine the target segment if the priority corresponding to the scene identifier is 1, locate the target segment in the first database cluster, and traverse the integer set structure in the target segment and subsequent non-empty segments to extract the first material identifier stored therein;

[0195] If the priority is 0 and the hot tag is 1, then read the scene identifier-material identifier list stored as a continuous integer array from the hot data area of ​​the database layer buffer pool to obtain the second material identifier set;

[0196] If the priority is 0 and the hot tag is 0, the scenario is excluded from the current candidate set construction or delayed until subsequent on-demand access.

[0197] The first material identifier and the second material identifier are merged, and global deduplication is performed to obtain a set of candidate material identifiers.

[0198] In an optional implementation of this embodiment, the retrieval result determination module 440 is specifically used to sort the candidate material identifiers according to the similarity calculation results, and to adjust the sorting results by combining scene priority and hot tag, and select a preset number of material identifiers with high sorting as target candidate material identifiers.

[0199] Based on the target candidate material identifier, the corresponding original material content is obtained from the metadata area of ​​the vectorized main library, and the original material content is returned as the retrieval result of this retrieval request.

[0200] The vector retrieval device for multiple service scenarios provided in the embodiments of the present invention can execute the vector retrieval method for multiple service scenarios provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0201] In the technical solutions of this invention, the collection, storage, use, processing, transmission, provision and disclosure of materials (such as content, identifiers, etc.) in different business scenarios all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0202] Example 4

[0203] Figure 5 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0204] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from the storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0205] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0206] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as vector retrieval methods for multi-service scenarios.

[0207] In some embodiments, the vector retrieval method for multiple service scenarios can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the vector retrieval method for multiple service scenarios described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the vector retrieval method for multiple service scenarios by any other suitable means (e.g., by means of firmware).

[0208] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0209] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0210] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM), optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0211] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0212] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0213] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and Virtual Private Servers (VPS) in terms of management difficulty and weak business scalability.

[0214] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0215] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

[0216] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements a database detection method as provided in any embodiment of this application.

[0217] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including LANs or WANs—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0218] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, they do not mean that the solution has been or necessarily used.

[0219] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A method for vector retrieval of multi-service scenarios, executed by a data center service, characterized in that, The method comprises: receiving a target retrieval request, determining a modal type of the target retrieval request, and determining a target retrieval vector matching the target retrieval request based on the modal type; determining a retrieval range in a predetermined target material library, and determining a candidate material identification set based on the retrieval range; wherein the target material library comprises materials of multiple business scenarios and feature vectors corresponding to each of the materials; determining the similarity of the target retrieval vector and the feature vector of each candidate material in the candidate material identification set; based on the similarity calculation result, determining a target candidate material identification, and determining the material content corresponding to the target candidate material identification as the retrieval result of the target retrieval request.

2. The method of claim 1, wherein, The determination of the modal type of the target retrieval request and the determination of the target retrieval vector matching the target retrieval request based on the modal type comprise: modal recognition of the target retrieval request to determine the modal type of the target retrieval request; determining a target encoding model matching the target retrieval request based on the modal type, and processing the target retrieval request based on the target encoding model to obtain a target retrieval vector matching the target retrieval request; wherein the modal type comprises at least one of the following: text, image, video and audio.

3. The method of claim 1, wherein, The target material library is determined based on the following steps: obtaining a material set of different business scenarios; wherein the material set comprises at least one of the following data: text data, image data, video data and audio data; based on the content information of each material in the material set, the scene identification and the unique material identification corresponding to each material are determined respectively; based on the scene identification of the target material, querying the preset scene table to obtain the priority information of the business scenario corresponding to the scene identification; based on the target material identification of the target material, determining the target shard in the target material identification corresponding to the target material in the first database cluster; determining whether there is a reference material identification in the target shard that is the same as the target material identification of the target material; if there is, the target material is determined as a duplicate material; if not, based on the target material and the target material identification, a feature vector of a fixed dimension is generated, and the target material identification and the feature vector form a target key-value pair; storing the target key-value pair in a vector storage area and storing the attribute information of the target material in a metadata area; the attribute information includes format, size or type; writing the target material identification into the target shard, and if the number of elements of the target shard reaches an upper limit value, writing the target material identification into the next unfulfilled shard; obtaining the target material library in response to a write completion instruction of the material identification of all materials.

4. The method of claim 3, wherein, Before obtaining the target material library in response to the write completion instruction of the material identification of all materials, the method further comprises: in response to a calling instruction of a target interface, querying the scene table according to the scene identification to obtain the priority information of the business scenario; If the priority is high, a target shard is determined, and the target shard is located in the first database cluster, and the material identification is written into the integer set of the shard; If the priority is low, the association between the scene identification and the material identification is written into the cold data area of the database layer buffer pool and is persisted to the scene identification-material identification list of the database; In response to a start instruction of the data center service, if it is detected that the preset path exists a buffer pool mirror file, the scene identification-material identification list saved in the mirror file is loaded to the hot data area of the buffer pool; if not, the associated data of the high-priority and low-priority scenes marked as hot is read from the database and loaded to the first database cluster and the hot data area of the buffer pool, respectively; In response to a close instruction of the data center service, the scene identification-material identification list currently stored in the hot data area of the buffer pool is serialized and written to the preset path, and a buffer pool mirror file is generated or updated.

5. The method of claim 4, wherein, The method further includes: determining a target retrieval scene corresponding to the target retrieval request, and determining the retrieval range based on the target retrieval scene; and determining a candidate material identification set based on priority marking information and hot marking information of each scene identification in the retrieval range.

6. The method of claim 5, wherein, The method further includes: if the priority corresponding to the scene identification is 1, determining a target shard, locating the target shard in the first database cluster, and traversing an integer set structure in the target shard and subsequent non-empty shards to extract a first material identification stored therein; if the priority is 0 and the hot marking is 1, reading a scene identification-material identification list stored in the form of a continuous integer array from the hot data area of the database layer buffer pool to obtain a second material identification set; if the priority is 0 and the hot marking is 0, excluding the scene from the current candidate set construction or delaying loading until subsequent on-demand access; merging the first material identification and the second material identification, and performing global deduplication processing to obtain the candidate material identification set.

7. The method of claim 1, wherein, The method further includes: sorting the candidate material identifications according to the similarity calculation result, and performing weighted adjustment on the sorting result in combination with scene priority and hot marking to select a preset number of material identifications with high ranking as target candidate material identifications; obtaining corresponding original material content from the metadata area of the vectorization master library according to the target candidate material identification, and returning the original material content as the retrieval result of the current retrieval request.

8. An electronic device, comprising: The electronic device comprises at least one processor and a memory connected with the at least one processor in communication; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the vector retrieval method for multi-business scenarios according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for enabling the processor to implement the vector retrieval method for multi-business scenarios according to any one of claims 1-7 when executed.

10. A computer program product, characterised in that, The computer program is executed by the processor to implement the vector retrieval method for multi-business scenarios according to any one of claims 1-7.

Images