Material recommendation method and system based on milvus vector similarity searching, and storage medium

Through the Milvus vector similarity search method, the problem of difficult to search for materials that are most similar to those of users clicking on materials in the prior art is solved, and more accurate material recommendations are achieved, improving the performance and effect of the recommendation algorithm.

WO2024229752A1PCT designated stage expired Publication Date: 2024-11-14SHANGHAI 2345 NETWORK TECH
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2023/093265
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-05
Filing Date
2023-05-10
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

It is difficult to search for materials that are most similar to those clicked by users, resulting in large differences in recommendation indicators such as revenue, click-through rate and arpu value from the actual situation of users.

Method used

Using the Milvus vector similarity search method, the embedded vector data of material instances is synchronized to the cloud native vector database cluster through the dolphin scheduling module, and the routing service module obtains a healthy IP list. The vector service module performs similar search for polling load balancing strategies. , calculate the cosine similarity, return the embedding vectors of the K material instances with the highest similarity and make recommendations.

Benefits of technology

Effectively searching for materials that are most similar to the material to be searched and materials that are most similar to the material instances that users click, improves the various indicators of the recommendation algorithm and meets multiple recommendation scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2023093265_14112024_PF_FP_ABST
    Figure CN2023093265_14112024_PF_FP_ABST
Patent Text Reader

Abstract

Provided in the present invention is a material recommendation method based on Milvus vector similarity searching. The method comprises the following method steps: S1, a dolphin scheduling module synchronizing embedding vector data of material instances to a cloud-native vector database cluster; S2, a routing service module acquiring an IP list of the material instances; S3, a vector service module acquiring the IP list of the material instances; S4, the vector service module requesting a plurality of cloud-native vector databases to perform similarity searching on embedding vectors of corresponding material instances in the IP list of the material instances; S5, the plurality of cloud-native vector databases performing cosine similarity calculation on an embedding vector of a material instance to be searched and service vectors in the cloud-native vector databases, and returning to the vector service module K embedding vectors having the highest similarity; and S6, a recommendation service module performing material recommendation according to the embedding vectors of K material instances. By means of the present invention, a material most similar to a material to be searched can be found, thereby improving various indicators of algorithm recommendation.
Need to check novelty before this filing date? Find Prior Art

Description

Material recommendation method, system and storage medium based on Milvus vector similarity search

[0001] This application claims priority to the Chinese patent application number CN 202310499383.0 (Invention Title: Material Recommendation Method, System and Storage Medium Based on Milvus Vector Similarity Search) filed on May 6, 2023. Technical Field

[0002] The present invention relates to the field of data processing technology, and in particular to a material recommendation method, system and storage medium based on Milvus vector similarity search. Background Art

[0003] Milvus (a cloud-native vector database) is a cloud-native vector database that offers high availability, high performance, and easy scalability for real-time recall of massive vector data. Built on vector search libraries such as FAISS, Annoy, and HNSW, Milvus' core focus is on solving the problem of dense vector similarity retrieval. Building on these libraries, Milvus supports data partitioning and sharding, data persistence, incremental data ingestion, scalar-vector hybrid queries, and time travel. It also significantly optimizes vector search performance to meet the application needs of any vector search scenario.

[0004] Existing material recommendation methods find it difficult to search for materials that are most similar to the materials clicked by users. Various indicators of material recommendation, such as revenue, click-through rate, and ARPU (average revenue per user), are significantly different from the actual situation of users.

[0005] Summary of the Invention

[0006] The present invention provides a material recommendation method, system and storage medium based on Milvus vector similarity search to solve the technical problem in the prior art that it is difficult to search for the material most similar to the material clicked by the user.

[0007] One aspect of the present invention is to provide a material recommendation method based on Milvus vector similarity search, the material recommendation method comprising the following steps:

[0008] S1. The Dolphin Scheduling Module synchronizes the embedded vector data of the material instance to a cloud native vector database cluster, where the cloud native vector database cluster includes multiple cloud native vector databases.

[0009] S2, the routing service module, obtains the IP list of material instances from multiple cloud-native vector databases;

[0010] S3. The vector service module calls the routing service module and obtains the IP list of the material instance from the routing service module;

[0011] S4, the vector service module, uses a round-robin load balancing strategy to request multiple cloud-native vector databases for the material instance IP list and performs a similarity search on the embedded vectors of the corresponding material instances in the material instance IP list.

[0012] S5. Multiple cloud-native vector databases perform cosine similarity calculations on the embedded vectors of the material instances being searched and the business vectors in the cloud-native vector databases. The embedded vectors of the K material instances with the highest similarity are returned to the vector service module.

[0013] S6. The recommendation service module obtains the embedding vectors of K material instances and recommends materials based on the embedding vectors of the K material instances.

[0014] In a preferred embodiment, in step S1, the Dolphin Scheduling Module synchronizes the embedded vector data of the material instance to the cloud native vector database cluster by calling the cloud native vector database data synchronization interface.

[0015] In a preferred embodiment, in step S3, the vector service module calls the routing service module through the HTTP protocol and obtains the IP list of the material instance from the routing service module.

[0016] In a preferred embodiment, in step S2, the routing service module performs a health check on multiple cloud native vector databases.

[0017] When unhealthy material instance IPs exist in multiple cloud native vector databases, the routing service module removes the unhealthy material instance IPs from the material instance IP list.

[0018] In a preferred embodiment, the vector service module obtains a material instance IP list excluding unhealthy material instance IPs from the routing service module.

[0019] In a preferred embodiment, in step S6, the recommendation service module calls the vector service module through the HTTP protocol, obtains the embedding vectors of K material instances from the vector service module, and recommends the materials corresponding to the embedding vectors of the K material instances.

[0020] Another aspect of the present invention is to provide a material recommendation system based on Milvus vector similarity search, the material recommendation system comprising:

[0021] A Dolphin scheduling module, configured to store the embedded vector data of a material instance and synchronize the embedded vector data of the material instance to a cloud native vector database cluster, wherein the cloud native vector database cluster includes multiple cloud native vector databases;

[0022] The routing service module is used to obtain the IP list of material instances from multiple cloud native vector databases;

[0023] The vector service module is used to call the routing service module to obtain the IP list of material instances from the routing service module. It is also used to request multiple cloud-native vector databases through a polling load balancing strategy for the material instance IP list and perform similarity searches on the embedded vectors of the corresponding material instances in the material instance IP list.

[0024] The cloud-native vector database cluster uses multiple cloud-native vector databases to perform cosine similarity calculations on the embedding vectors of the searched material instances and the business vectors in the cloud-native vector databases. It then returns the embedding vectors of the K material instances with the highest similarity to the vector service module.

[0025] The recommendation service module is used to obtain the embedding vectors of K material instances and make material recommendations based on the embedding vectors of the K material instances.

[0026] In a preferred embodiment, the material recommendation system further includes a cloud-native vector database data synchronization interface;

[0027] The Dolphin Scheduling module synchronizes the embedded vector data of the material instance to the cloud native vector database cluster by calling the cloud native vector database data synchronization interface.

[0028] In a preferred embodiment, the routing service module is also used to perform health checks on multiple cloud native vector databases.

[0029] When unhealthy material instance IPs exist in multiple cloud native vector databases, the routing service module removes the unhealthy material instance IPs from the material instance IP list.

[0030] In another embodiment of the present invention, a computer storage medium is provided, wherein the computer storage medium is used to store computer execution instructions, and the computer execution instructions are used to execute a material recommendation method based on Milvus vector similarity search of the present invention.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] The present invention provides a material recommendation method, system and storage medium based on Milvus vector similarity search. Based on Milvus vector similarity search, it can search for materials that are most similar to the material to be searched, as well as materials that are most similar to the material instance clicked by the user, thereby improving various indicators of algorithm recommendation and satisfying multiple recommendation algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0034] FIG1 is a flow chart of a material recommendation method based on Milvus vector similarity search according to the present invention.

[0035] FIG2 is a structural block diagram of a material recommendation system based on Milvus vector similarity search according to the present invention. DETAILED DESCRIPTION

[0036] In order to make the above and other features and advantages of the present invention more clear, the present invention is further described below with reference to the accompanying drawings. It should be understood that the specific embodiments given herein are for the purpose of explaining to those skilled in the art and are only exemplary and not restrictive.

[0037] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0038] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0039] FIG1 is a flowchart of a material recommendation method based on Milvus vector similarity search according to the present invention, and FIG2 is a structural block diagram of a material recommendation system based on Milvus vector similarity search according to the present invention. According to an embodiment of the present invention, a material recommendation method based on Milvus vector similarity search is provided, including the following method steps:

[0040] In step S1, the dolphin scheduling module (dolphinscheduler) 100 synchronizes the embedding data of the material instance to the cloud native vector database cluster 300, wherein the cloud native vector database cluster 300 includes multiple cloud native vector databases (Milvus).

[0041] The materials in the present invention can be text news, video news, applications, games, etc. For example, the materials come from: Dajiang News, Kuniu News, Oriental News, Taiwan Strait News, Douyin short videos, etc.

[0042] A material instance refers to a user's actions on an item, such as a click. The cloud-native vector database (Milvus) stores the IP address list of the material instance and the business vector library. The IP address list of the material instance uses the material instance's embedding vector as the title.

[0043] The embedding vectors of material instances are all 128-dimensional floating-point arrays. For example, the embedding vectors of material instances are:

[0044] [0.02346399985253811,-0.19658300280570984,-0.14361600577831268,-0.10559000074863434,-0.01370800007134676,-0.09146100282669067,-0.026830999180674553,-0.06744299829006195,0.1114...]

[0045] The cloud native vector database cluster 300 includes multiple cloud native vector databases (Milvus), for example, Milvus 1, Milvus 2, Milvus 3, ..., Milvus N.

[0046] According to an embodiment of the present invention, the Dolphin scheduling module 100 synchronizes the embedded vector data of the material instance to the cloud native vector database cluster 300 by calling the cloud native vector database data synchronization interface 200 .

[0047] In a preferred embodiment, the embedded vector data of the material instance synchronized by the Dolphin scheduling module 100 to the cloud native vector database cluster 300 includes full vector data and incremental vector data.

[0048] Step S2: The routing service module 400 obtains the IP list of the material instance from multiple cloud native vector databases (Milvus).

[0049] Step S3: The vector service module 500 calls the routing service module 400 and obtains the IP list of the material instance from the routing service module 400.

[0050] In one embodiment, in step S2, the routing service module 400 performs a health check on multiple cloud native vector databases (Milvus). When there are unhealthy material instance IPs in the multiple cloud native vector databases (Milvus), the routing service module 400 removes the unhealthy material instance IPs from the material instance IP list.

[0051] The vector service module 500 calls the routing service module 400 via HTTP protocol and obtains the IP list of the material instance from the routing service module 400. Furthermore, the vector service module 500 obtains the material instance IP list from the routing service module 400, excluding unhealthy material instance IPs.

[0052] Step S4: Perform similarity search on the embedding vector of the corresponding material instance in the material instance IP list.

[0053] The vector service module 500 uses a round-robin load balancing strategy to request multiple cloud native vector databases (Milvus) for the material instance IP list, and performs a similarity search on the embedded vectors of the corresponding material instances in the material instance IP list.

[0054] Step S5: The embedding vectors of the K material instances with the highest similarity are returned to the vector service module 500.

[0055] Multiple cloud native vector databases (Milvus) perform cosine similarity calculation on the embedding vectors of the material instances to be searched and the business vectors in the cloud native vector database (Milvus), and return the embedding vectors of the K material instances with the highest similarity to the vector service module 500.

[0056] Step S6: Material recommendation.

[0057] The recommendation service module 600 obtains the embedding vectors of K material instances and recommends materials based on the embedding vectors of the K material instances.

[0058] Specifically, the recommendation service module 600 calls the vector service module 500 through the HTTP protocol, obtains the embedding vectors of the K material instances with the highest similarity from the vector service module 500, and recommends the materials corresponding to the embedding vectors of the K material instances.

[0059] As shown in FIG2 , according to an embodiment of the present invention, a material recommendation system based on Milvus vector similarity search is provided, including:

[0060] The dolphin scheduling module (dolphinscheduler) 100 is used to store the embedded vector data of the material instance and synchronize the embedded vector data of the material instance to the cloud native vector database cluster 300. The cloud native vector database cluster 300 includes multiple cloud native vector databases (Milvus).

[0061] The cloud native vector database data synchronization interface 200, the Dolphin scheduling module 100 synchronizes the embedded vector data of the material instance to the cloud native vector database cluster 300 by calling the cloud native vector database data synchronization interface 200.

[0062] The routing service module 400 is used to obtain the IP list of the material instance from multiple cloud native vector databases (Milvus).

[0063] The routing service module 400 also performs health checks on multiple cloud native vector databases (Milvus). When there are unhealthy material instance IPs in multiple cloud native vector databases (Milvus), the routing service module 400 removes the unhealthy material instance IPs from the material instance IP list.

[0064] The vector service module 500 is used to call the routing service module 400, obtain the IP list of the material instance from the routing service module 400, and use the polling load balancing strategy to request multiple cloud native vector databases (Milvus) to perform similarity search on the embedded vectors of the corresponding material instances in the material instance IP list.

[0065] The cloud native vector database cluster 300, through multiple cloud native vector databases (Milvus), performs cosine similarity calculation on the embedding vector of the material instance to be searched and the business vector in the cloud native vector database (Milvus), and returns the embedding vectors of the K material instances with the highest similarity to the vector service module 500.

[0066] The recommendation service module 600 is used to obtain the embedding vectors of K material instances and make material recommendations based on the embedding vectors of the K material instances.

[0067] According to an embodiment of the present invention, a computer storage medium is provided for storing computer-executable instructions for executing a material recommendation method based on Milvus vector similarity search provided by the present invention.

[0068] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A material recommendation method based on Milvus vector similarity search, characterized in that: The material recommendation method comprises the following steps: S1. The Dolphin scheduling module synchronizes the embedded vector data of the material instance to the cloud native vector database cluster, wherein the cloud native vector database cluster includes multiple cloud native vector databases; S2, the routing service module, obtains the IP list of material instances from multiple cloud native vector databases; S3. The vector service module calls the routing service module and obtains the IP list of the material instance from the routing service module; S4, the vector service module, requests multiple cloud-native vector databases through a polling load balancing strategy for the material instance IP list, and performs similarity search on the embedded vectors of the corresponding material instances in the material instance IP list; S5. Multiple cloud-native vector databases perform cosine similarity calculation on the embedded vectors of the material instances to be searched and the business vectors in the cloud-native vector database, and return the embedded vectors of the K material instances with the highest similarity to the vector service module; S6: The recommendation service module obtains the embedding vectors of the K material instances and recommends materials based on the embedding vectors of the K material instances.

2. The material recommendation method according to claim 1, characterized in that: In step S1, the Dolphin Scheduling Module synchronizes the embedded vector data of the material instance to the cloud native vector database cluster by calling the cloud native vector database data synchronization interface.

3. The material recommendation method according to claim 1, characterized in that: In step S3, the vector service module calls the routing service module through the HTTP protocol and obtains the IP list of the material instance from the routing service module.

4. The material recommendation method according to claim 1, characterized in that: In step S2, the routing service module performs health checks on multiple cloud native vector databases. When there are unhealthy material instance IPs in multiple cloud native vector databases, the routing service module removes the unhealthy material instance IPs from the material instance IP list.

5. The material recommendation method according to claim 4, characterized in that: The vector service module obtains the material instance IP list excluding unhealthy material instance IPs from the routing service module.

6. The material recommendation method according to claim 1, characterized in that: In step S6, the recommendation service module calls the vector service module through the HTTP protocol, obtains the embedding vectors of the K material instances from the vector service module, and recommends the materials corresponding to the embedding vectors of the K material instances.

7. A material recommendation system based on Milvus vector similarity search, characterized in that: The material recommendation system comprises: A dolphin scheduling module, used to store the embedded vector data of the material instance and synchronize the embedded vector data of the material instance to a cloud native vector database cluster, wherein the cloud native vector database cluster includes multiple cloud native vector databases; The routing service module is used to obtain the IP list of material instances from multiple cloud native vector databases; The vector service module is used to call the routing service module, obtain the IP list of the material instance from the routing service module, and request multiple cloud native vector databases through the load balancing strategy of polling the IP list of the material instance to perform similarity search on the embedded vector of the corresponding material instance in the IP list of the material instance; The cloud-native vector database cluster uses multiple cloud-native vector databases to perform cosine similarity calculations on the embedding vectors of the material instances being searched and the business vectors in the cloud-native vector databases, and returns the embedding vectors of the K material instances with the highest similarity to the vector service module. The recommendation service module is used to obtain the embedding vectors of K material instances and make material recommendations based on the embedding vectors of the K material instances.

8. The material recommendation system according to claim 7, characterized in that: The material recommendation system also includes a cloud native vector database data synchronization interface; The Dolphin Scheduling Module synchronizes the embedded vector data of the material instance to the cloud native vector database cluster by calling the cloud native vector database data synchronization interface.

9. The material recommendation system according to claim 7, characterized in that: The routing service module is also used to perform health checks on multiple cloud native vector databases. When there are unhealthy material instance IPs in multiple cloud native vector databases, the routing service module removes the unhealthy material instance IPs from the material instance IP list.

10. A computer storage medium, characterized in that: The computer storage medium is used to store computer executable instructions, and the computer executable instructions are used to execute the material recommendation method described in any one of claims 1 to 6.

Citation Information

Patent Citations

  • Industrial material commodity recommendation method

    CN107123016A

  • Recommended material determination method and device, storage medium and computer equipment

    CN109408729A

  • Material recommendation method in enterprise WeChat and system thereof

    CN112989191A

  • Article recommendation method and system, electronic equipment and storage medium

    CN113704617A

  • System, apparatus and method for providing weight to information gathering engine according to situation of user and computer readable medium processing the method

    US20080147635A1