Vector engineering recommendation method
Through shared storage, storage standardization, configurable vector services and XXL-JOB technology, the problem of low performance of ES clusters is solved, and efficient vector data processing and performance improvement of recommendation systems is achieved.
Patent Information
- Application Number
- CN202510037438.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-09
- Publication Date
- 2025-04-25
AI Technical Summary
In the existing technology, the ES cluster is huge in scale and occupies a lot of resources. The ES-based vector query service has poor performance and high maintenance costs. Each time a new index is added, the entire process is required, the development cycle is long, the historical data is redundant, and the mechanism for regularly deleting and cleaning useless data is lacking.
It adopts a shared storage mechanism, storage standardized design, and vector services to support configurability, regular inspection and deletion of vector files, and combines XXL-JOB technology to conduct regular inspections and cleaning, and provides HTTP and gRPC services to support vector query.
It improves the storage and computing efficiency of vector data, improves the accuracy of recommendation systems, reduces maintenance costs, shortens development cycles, reduces historical data redundancy, and supports dynamic vector updates and efficient vector queries.
Smart Images
Figure CN119884123A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a vector engineering recommendation method and belongs to the technical field of vector engineering recommendation applications. Background Art
[0002] The existing technology uses ES's vector search for real-time vector queries in recommendation systems. The project creates an index, the algorithm writes index data through scheduled task scheduling, and the engineering staff writes DSL statements to provide online vector search and recall relevant material data.
[0003] It can be seen that the existing technology has the following shortcomings: the ES cluster is large in scale and occupies a lot of resources. The performance of the vector query service based on ES is poor. At present, 99 lines in the system are above 90, the maintenance cost is high, and the interaction with the algorithm is complex. Every time a new index is added, the entire process needs to be followed, the development cycle is long, and there is a lot of historical data redundancy. There is no regular deletion and cleaning of useless data. Compared with the current mainstream vector search engines, such as Milvus, Faiss, Vsearch, Qdrant, ProximaSE, ProximaBEMilvus, Vsearch, and Qdrant support distributed and stand-alone deployment, and support multiple index types, including but not limited to FLAT, IVF, HNSW, etc., to meet different search needs. However, the application in online recommendation systems has high performance requirements and cannot meet the performance requirements for the time being. Faiss provides fast retrieval for large-scale vector data by using efficient memory indexing algorithms such as IVF and PQ. It is particularly suitable for scenarios that need to process billions of vectors. Faiss currently supports calls from C++, Python, and GO, and is not friendly to JAVA. ProximaSE supports scalar and vector queries, supports multiple index types, supports distributed deployment, and its performance meets the requirements of recommendation systems. However, the SE version is not open source yet and only has a commercial version. ProximaBE does not support scalar data queries, but supports vector data queries, single-machine deployment, and supports millisecond-level queries on billions of vectors. The project is open source and meets the performance requirements of recommendation systems. Summary of the invention
[0004] In this embodiment, a vector engineering recommendation method is provided to solve the problems in the prior art that the ES cluster is large in scale, occupies a lot of resources, and the performance of the vector query service based on ES is poor. At present, 99 lines in the system are more than 90, the maintenance cost is high, and the interaction with the algorithm is complex. Each time a new index is added, the entire process needs to be followed, the development cycle is long, and there is a lot of historical data redundancy. There is no regular deletion and cleaning of useless data. Compared with the current mainstream vector retrieval engines, such as Milvus, Faiss, Vsearch, Qdrant, ProximaSE, ProximaBEMilvus, Vsearch, and Qdrant support distributed and stand-alone deployment, and support multiple index types, including but not limited to FLAT, IVF, HNSW, etc., to meet different search needs. However, the application in online recommendation systems has high performance requirements and cannot meet the performance requirements for the time being. Faiss provides fast retrieval for large-scale vector data by using efficient memory indexing algorithms such as IVF and PQ. It is particularly suitable for scenarios that need to process billions of vectors. Faiss currently supports calls from C++, Python, and GO, and is not friendly to JAVA. ProximaSE supports scalar and vector queries, supports multiple index types, supports distributed deployment, and its performance meets the requirements of recommendation systems. However, the SE version is not open source yet and only has a commercial version. ProximaBE does not support scalar data queries, but supports vector data queries, stand-alone deployment, supports millisecond-level queries on billions of vectors, and the project is open source, which meets the performance requirements of recommendation systems.
[0005] The present invention achieves the above-mentioned object through the following technical scheme, a vector engineering recommendation method, and the vector engineering recommendation method comprises the following steps:
[0006] S1, shared storage mechanism, using a shared storage system to ensure that multiple servers can access consistent data content and achieve data synchronization and sharing;
[0007] S2. Storage standardization design includes directory structure standardization, file naming standardization and file content format standardization;
[0008] S3 and vector services support configurability. The service loads the current vector file data according to the configuration path, builds a vector collection according to fieldList, sets the vector field and vector dimension according to vectorFieldName and dimensions, and uses vectorName as the name of the created vector collection. Through these configurations, you can create a vector collection and load vector collection data.
[0009] S4. Check and update the vector regularly. The service integrates XXL-JOB and uses the job to regularly check whether the vector file has been updated. During the update process, the number of records of the current vector will be compared and recorded. The update status of the vector can be intuitively seen through the job log.
[0010] S5. Regularly delete historical vectors. The service integrates XXL-JOB and uses jobs to regularly clean up historical vectors to ensure the stability of the vector server.
[0011] S6. Provide external services. Vector Engineering provides easy-to-use Http services and Grpc services to facilitate the calling of other services.
[0012] Preferably, the shared storage mechanism in step (S1) refers to a technology and method that allows multiple servers, virtual machines or clients to concurrently access and use the same storage device or file system in a computer network or cluster environment. The shared storage mechanism is applied in scenarios of large-scale computing, virtualization, databases, high availability clusters (HA), and distributed computing.
[0013] Preferably, the directory structure normalization in step (S2) includes setting a unified vector file storage path, and all vectors are stored in the / data / model / share / feature / vector directory.
[0014] Preferably, the file naming standardization in step (S2) includes using .txt.conf as a suffix for all files to clearly identify the file type.
[0015] Preferably, the standardized file content format in step (S2) includes type_material id, update time, and vector content, separated by a space symbol.
[0016] Preferably, the Job log in the step (S4) refers to a record file automatically generated by the system or application during the execution of the job, which is used to record the execution status, output results, error information and other related information of the job.
[0017] Preferably, in the step (S6), HTTP, namely Hypertext Transfer Protocol, is the most widely used network protocol on the Internet. The HTTP service adopts a request / response model, that is, the client sends a request to the server, and the server responds with a status line. The default port of the HTTP service is 80, and the default port of HTTPS (HTTP Secure) is 443. The latter is a secure version of HTTP, which transmits data through SSL / TLS encryption.
[0018] Preferably, the gRPC service in the step (S6) is a high-performance, open source and general remote procedure call (RPC) framework developed by Google, which allows client and server applications to communicate in a transparent manner, just like calling local object methods. Calling remote methods, gRPC is designed based on the HTTP / 2 protocol and uses Protocol Buffers (Protobuf for short) as its interface definition language (IDL) and serialization tool.
[0019] Preferably, in the step (S3), fieldList refers to a list of fields involved, and fieldList is used to specify the fields required to be operated when performing a database query or update operation. These fields are the basic building blocks of the data table and store specific data information.
[0020] Preferably, in the step (S3), vectorName generally refers to the name of a vector object. Vector is a sequence container in STL that can store a variable number of elements of the same type. These elements are stored consecutively in memory. Vector provides a random access iterator that allows quick access to any element through a subscript. It also provides many member functions to operate these elements, such as adding, deleting, searching, etc.
[0021] The beneficial effects of the present invention are as follows: the vector engineering recommendation method of the present invention solves the storage and vector calculation of large-scale or high-dimensional vector data, improves efficiency, improves the accuracy of the recommendation system, better captures user preferences and content characteristics, standardizes vector interaction, solves the dynamic vector update mechanism and its application in the recommendation system, provides a unified and configurable vector retrieval service, dynamically perceives the vector update mechanism, and supports external vector query services of multiple protocols;
[0022] Provides efficient vector query services: By integrating the open source vector engine and combining the existing cache technology and XXL-JOB technology, it provides reliable vector query services, which can perform millisecond-level queries on millions of vectors.
[0023] Configurable vectors: Use configuration files to configure the vector set to be generated, which improves the efficiency of vector engineering development;
[0024] Greatly reduced costs: Previously, we used an ES cluster as the engine for vector queries. The 13 high-configuration machines had a total of 112.0 CPU cores, 230.0 GB memory, and 1400.0 GB disk. Now, we have reduced the number of machines to a single 32-core 64 GB virtual machine, which has higher performance and is more stable than the ES cluster.
[0025] Support dynamic update: use XXL-JOB to regularly check the vector file configuration path, use the file MD5 check to check whether the file is updated, support abnormal file check, if the number of vectors does not meet the configured number of strips, it will not be updated. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION
[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0028] Embodiment 1:
[0029] A vector engineering recommendation method, the vector engineering recommendation method comprising the following steps:
[0030] S1, shared storage mechanism, using a shared storage system to ensure that multiple servers can access consistent data content and achieve data synchronization and sharing;
[0031] S2. Storage standardization design includes directory structure standardization, file naming standardization and file content format standardization;
[0032] S3 and vector services support configurability. The service loads the current vector file data according to the configuration path, builds a vector collection according to fieldList, sets the vector field and vector dimension according to vectorFieldName and dimensions, and uses vectorName as the name of the created vector collection. Through these configurations, you can create a vector collection and load vector collection data.
[0033] S4. Check and update the vector regularly. The service integrates XXL-JOB and uses the job to regularly check whether the vector file has been updated. During the update process, the number of records of the current vector will be compared and recorded. The update status of the vector can be intuitively seen through the job log.
[0034] S5. Regularly delete historical vectors. The service integrates XXL-JOB and uses jobs to regularly clean up historical vectors to ensure the stability of the vector server.
[0035] S6. Provide external services. Vector Engineering provides easy-to-use Http services and Grpc services to facilitate the calling of other services.
[0036] Preferably, the shared storage mechanism in step (S1) refers to a technology and method that allows multiple servers, virtual machines or clients to concurrently access and use the same storage device or file system in a computer network or cluster environment. The shared storage mechanism is applied in scenarios of large-scale computing, virtualization, databases, high availability clusters (HA), and distributed computing.
[0037] Preferably, the directory structure normalization in step (S2) includes setting a unified vector file storage path, and all vectors are stored in the / data / model / share / feature / vector directory.
[0038] Preferably, the file naming standardization in step (S2) includes using .txt.conf as a suffix for all files to clearly identify the file type.
[0039] Preferably, the standardized file content format in step (S2) includes type_material id, update time, and vector content, separated by a space symbol.
[0040] Preferably, the Job log in the step (S4) refers to a record file automatically generated by the system or application during the execution of the job, which is used to record the execution status, output results, error information and other related information of the job.
[0041] Preferably, in the step (S6), HTTP, namely Hypertext Transfer Protocol, is the most widely used network protocol on the Internet. The HTTP service adopts a request / response model, that is, the client sends a request to the server, and the server responds with a status line. The default port of the HTTP service is 80, and the default port of HTTPS (HTTP Secure) is 443. The latter is a secure version of HTTP, which transmits data through SSL / TLS encryption.
[0042] Preferably, the gRPC service in the step (S6) is a high-performance, open source and general remote procedure call (RPC) framework developed by Google, which allows client and server applications to communicate in a transparent manner, just like calling local object methods. Calling remote methods, gRPC is designed based on the HTTP / 2 protocol and uses Protocol Buffers (Protobuf for short) as its interface definition language (IDL) and serialization tool.
[0043] Preferably, in the step (S3), fieldList refers to a list of fields involved, and fieldList is used to specify the fields required to be operated when performing a database query or update operation. These fields are the basic building blocks of the data table and store specific data information.
[0044] Preferably, in the step (S3), vectorName generally refers to the name of a vector object. Vector is a sequence container in STL that can store a variable number of elements of the same type. These elements are stored consecutively in memory. Vector provides a random access iterator that allows quick access to any element through a subscript. It also provides many member functions to operate these elements, such as adding, deleting, searching, etc.
[0045] The above-mentioned vector engineering recommendation method solves the storage and vector calculation of large-scale or high-dimensional vector data, improves efficiency, improves the accuracy of the recommendation system, better captures user preferences and content characteristics, standardizes vector interactions, solves the dynamic vector update mechanism and its application in the recommendation system, provides a unified and configurable vector retrieval service, dynamically perceives the vector update mechanism, and supports external vector query services of multiple protocols; the cost is greatly reduced: previously, the ES cluster was used as the engine for vector query, with 13 high-configuration machines with a total of 112.0 CPU cores, 230.0G memory, and 1400.0G disk, which were reduced to a single-machine 32-core 64G virtual machine, and the performance is higher and more stable than the ES cluster; supports dynamic updates: uses XXL-JOB to regularly check the vector file configuration path, uses file MD5 verification to compare whether the file is updated, supports abnormal file inspection, and does not update if the number of vectors does not meet the configured number of entries.
[0046] Embodiment 2:
[0047] A vector engineering recommendation method, the vector engineering recommendation method comprising the following steps:
[0048] S1, shared storage mechanism, using a shared storage system to ensure that multiple servers can access consistent data content and achieve data synchronization and sharing;
[0049] S2. Storage standardization design includes directory structure standardization, file naming standardization and file content format standardization;
[0050] S3 and vector services support configurability. The service loads the current vector file data according to the configuration path, builds a vector collection according to fieldList, sets the vector field and vector dimension according to vectorFieldName and dimensions, and uses vectorName as the name of the created vector collection. Through these configurations, you can create a vector collection and load vector collection data.
[0051] S4. Check and update the vector regularly. The service integrates XXL-JOB and uses the job to regularly check whether the vector file has been updated. During the update process, the number of records of the current vector will be compared and recorded. The update status of the vector can be intuitively seen through the job log.
[0052] S5. Regularly delete historical vectors. The service integrates XXL-JOB and uses jobs to regularly clean up historical vectors to ensure the stability of the vector server.
[0053] S6. Provide external services. Vector Engineering provides easy-to-use Http services and Grpc services to facilitate the calling of other services.
[0054] Preferably, the shared storage mechanism in step (S1) refers to a technology and method that allows multiple servers, virtual machines or clients to concurrently access and use the same storage device or file system in a computer network or cluster environment. The shared storage mechanism is applied in scenarios of large-scale computing, virtualization, databases, high availability clusters (HA), and distributed computing.
[0055] Preferably, the directory structure normalization in step (S2) includes setting a unified vector file storage path, and all vectors are stored in the / data / model / share / feature / vector directory.
[0056] Preferably, the file naming standardization in step (S2) includes using .txt.conf as a suffix for all files to clearly identify the file type.
[0057] Preferably, the standardized file content format in step (S2) includes type_material id, update time, and vector content, separated by a space symbol.
[0058] Preferably, the Job log in the step (S4) refers to a record file automatically generated by the system or application during the execution of the job, which is used to record the execution status, output results, error information and other related information of the job.
[0059] Preferably, in the step (S6), HTTP, namely Hypertext Transfer Protocol, is the most widely used network protocol on the Internet. The HTTP service adopts a request / response model, that is, the client sends a request to the server, and the server responds with a status line. The default port of the HTTP service is 80, and the default port of HTTPS (HTTP Secure) is 443. The latter is a secure version of HTTP, which transmits data through SSL / TLS encryption.
[0060] Preferably, the gRPC service in the step (S6) is a high-performance, open source and general remote procedure call (RPC) framework developed by Google, which allows client and server applications to communicate in a transparent manner, just like calling local object methods. Calling remote methods, gRPC is designed based on the HTTP / 2 protocol and uses Protocol Buffers (Protobuf for short) as its interface definition language (IDL) and serialization tool.
[0061] Preferably, in the step (S3), fieldList refers to a list of fields involved, and fieldList is used to specify the fields required to be operated when performing a database query or update operation. These fields are the basic building blocks of the data table and store specific data information.
[0062] Preferably, in the step (S3), vectorName generally refers to the name of a vector object. Vector is a sequence container in STL that can store a variable number of elements of the same type. These elements are stored consecutively in memory. Vector provides a random access iterator that allows quick access to any element through a subscript. It also provides many member functions to operate these elements, such as adding, deleting, searching, etc.
[0063] The above-mentioned vector engineering recommendation method solves the storage and vector calculation of large-scale or high-dimensional vector data, improves efficiency, improves the accuracy of the recommendation system, better captures user preferences and content characteristics, standardizes vector interactions, solves the dynamic vector update mechanism and its application in the recommendation system, provides a unified and configurable vector retrieval service, dynamically perceives the vector update mechanism, and supports external vector query services of multiple protocols; provides efficient vector query services: through the integration of open source vector engines and the combination of existing caching technology and XXL-JOB technology, it provides reliable vector query services, which can achieve millisecond-level queries on millions of vectors, and configurable vectors: using configuration files, configure the vector set to be generated, which improves the efficiency of vector engineering development; the cost is greatly reduced: previously, ES cluster was used as the engine for vector query, with 13 high-configuration machines with a total of 112.0 CPU cores, 230.0G memory, and 1400.0G disk, which were reduced to a single-machine 32-core 64G virtual machine, and the performance is higher and more stable than that of ES cluster.
[0064] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.
[0065] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.
Claims
1. A vector engineering recommendation method, characterized in that: The vector engineering recommendation method comprises the following steps: S1, shared storage mechanism, using a shared storage system to ensure that multiple servers can access consistent data content and achieve data synchronization and sharing; S2. Storage standardization design includes directory structure standardization, file naming standardization and file content format standardization; S3 and vector services support configurability. The service loads the current vector file data according to the configuration path, builds a vector collection according to fieldList, sets the vector field and vector dimension according to vectorFieldName and dimensions, and uses vectorName as the name of the created vector collection. Through these configurations, you can create a vector collection and load vector collection data. S4. Check and update the vector regularly. The service integrates XXL-JOB and uses the job to regularly check whether the vector file has been updated. During the update process, the number of records of the current vector will be compared and recorded. The update status of the vector can be intuitively seen through the job log. S5. Regularly delete historical vectors. The service integrates XXL-JOB and uses jobs to regularly clean up historical vectors to ensure the stability of the vector server. S6. Provide external services. Vector Engineering provides easy-to-use Http services and Grpc services to facilitate the calling of other services.
2. The vector engineering recommendation method according to claim 1, characterized in that: The shared storage mechanism in step (S1) refers to a technology and method that allows multiple servers, virtual machines or clients to concurrently access and use the same storage device or file system in a computer network or cluster environment. The shared storage mechanism is applied in scenarios of large-scale computing, virtualization, databases, high availability clusters (HA), and distributed computing.
3. The vector engineering recommendation method according to claim 1, characterized in that: The directory structure normalization in the step (S2) includes setting a unified vector file storage path, and all vectors are stored in the / data / model / share / feature / vector directory.
4. The vector engineering recommendation method according to claim 1, characterized in that: The file naming standardization in the step (S2) includes using .txt.conf as the suffix of all files to clearly identify the file type.
5. The vector engineering recommendation method according to claim 1, characterized in that: The file content format normalization in the step (S2) includes type_material id, update time, and vector content, which are separated by space symbols.
6. The vector engineering recommendation method according to claim 1, characterized in that: The job log in step (S4) refers to a record file automatically generated by the system or application during the execution of the job, which is used to record the execution status, output results, error information and other related information of the job.
7. The vector engineering recommendation method according to claim 1, characterized in that: In the step (S6), HTTP, namely Hypertext Transfer Protocol, is the most widely used network protocol on the Internet. The HTTP service adopts a request / response model, that is, the client sends a request to the server, and the server responds with a status line. The default port of the HTTP service is 80, and the default port of HTTPS (HTTP Secure) is 443. The latter is a secure version of HTTP, which transmits data through SSL / TLS encryption.
8. The vector engineering recommendation method according to claim 1, characterized in that: The gRPC service in the step (S6) is a high-performance, open source and general remote procedure call (RPC) framework developed by Google. It allows client and server applications to communicate in a transparent manner, just like calling local object methods. Calling remote methods, gRPC is designed based on the HTTP / 2 protocol and uses Protocol Buffers (Protobuf for short) as its interface definition language (IDL) and serialization tool.
9. The vector engineering recommendation method according to claim 1, characterized in that: In the step (S3), fieldList refers to a list of fields involved. FieldList is used to specify the fields required to be operated when performing database query or update operations. These fields are the basic building blocks of the data table and store specific data information.
10. The vector engineering recommendation method according to claim 1, characterized in that: In the step (S3), vectorName generally refers to the name of a vector object. Vector is a sequence container in STL that can store a variable number of elements of the same type. These elements are stored consecutively in memory. Vector provides a random access iterator that allows quick access to any element through a subscript. It also provides many member functions to operate these elements, such as adding, deleting, searching, etc.