Data processing method and apparatus, and computing device cluster

Abstract

A data processing method, comprising: acquiring first material content data and a first scalar filtering condition related to the first material content data; performing vectorization processing on the first material content data and the first scalar filtering condition by means of a neural network model, so as to obtain a first vector representation, wherein the first vector representation is used for representing the first material content data and the first scalar filtering condition; and loading the first vector representation into a vector index. The method can significantly improve the performance and effectiveness of vector search with scalar filtering conditions.

Description

A data processing method, apparatus, and computing device cluster

[0001] This application claims priority to Chinese Patent Application No. 202411219409.2, filed with the China National Intellectual Property Administration on August 31, 2024, entitled “A Data Processing Method, Apparatus and Computing Device Cluster”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI) technology, and in particular to a data processing method, apparatus and computing device cluster. Background Technology

[0003] With the widespread application of AI technology in data retrieval, search has evolved from keyword-based retrieval to vector retrieval based on data similarity. AI technology can transform not only structured data into multi-dimensional vectors, but also unstructured data. In vector retrieval, the distance between vectors is typically used to represent the similarity of corresponding data. However, user queries often contain scalar data information as filtering conditions in addition to vector data. This necessitates considering additional filtering conditions while capturing semantic similarity during the retrieval process. For example, in the legal field, it might only be necessary to search for relevant legal clauses from a specific tag database; in the retail industry, it might only be necessary to search for men's shoes in a specific size, and so on. This type of problem can be described as vector retrieval with scalar filtering conditions.

[0004] In related technologies, to achieve vector retrieval with scalar filtering conditions, scalar filtering is usually performed first, followed by vector retrieval (Pre-filter), or vector retrieval is performed first, followed by scalar filtering (Post-filter). However, these retrieval methods typically suffer from poor retrieval performance and effectiveness. Summary of the Invention

[0005] This application provides a data processing method, apparatus, computing device cluster, computer storage medium, and computer product, which can significantly improve the performance and effectiveness of vector retrieval with scalar filtering conditions.

[0006] In a first aspect, this application provides a data processing method, comprising: acquiring first material content data and a first scalar filtering condition related to the first material content data; performing vectorization processing on the first material content data and the first scalar filtering condition through a neural network model to obtain a first vector representation, wherein the first vector representation is used to characterize the first material content data and the first scalar filtering condition; and loading the first vector representation into a vector index.

[0007] In this way, by uniformly converting material content data and scalar filtering conditions into vector representations and constructing a vector index, it is no longer necessary to store the scalar filtering condition index separately, nor is it necessary to consider the number and complexity of scalar filtering conditions. This eliminates the need for additional external components, saving disk and memory storage space for the index and improving the latency of offline index construction. Furthermore, with this index structure, during the online retrieval phase, by using a unified input standard (i.e., both scalar filtering conditions and material content data are input as vector representations), and through a single retrieval, it is possible to simultaneously query and filter both material content data and scalar filtering conditions. This solves the problem of inconsistency between vector and scalar retrieval targets, reduces losses before and after scalar filtering, and improves retrieval accuracy. Moreover, because a unified input standard eliminates the need to consider the number and complexity of scalar filtering conditions during the retrieval phase, it solves the problem of increased overall latency and complexity in online retrieval, thus improving retrieval performance.

[0008] In one possible implementation, a neural network model is used to vectorize the first material content data and the first scalar filtering condition to obtain a first vector representation. This includes: vectorizing the first material content data to obtain a third vector representation; vectorizing the first scalar filtering condition to obtain a fourth vector representation; and performing interactive learning between the third and fourth vector representations to obtain the first vector representation. This allows the first material content data and the first scalar filtering condition to be integrated. Simultaneously, during the integration process, the first material content data and the first scalar filtering condition can interact, allowing both to learn from each other. This facilitates the grouping of similar types of ideas and scalars together when building the index.

[0009] In one possible implementation, the first material content data is vectorized to obtain a third vector representation. This includes: identifying the first material content data to obtain at least one first identifier; and performing interactive learning on the at least one first identifier to obtain the third vector representation. This converts the first material content data into a vector representation. Simultaneously, during the conversion process, the data within the first material content data can interact, allowing different data to learn from each other, improving the accuracy of the obtained vector representation, and facilitating the subsequent aggregation of similar material content data.

[0010] In one possible implementation, the first scalar filtering condition is vectorized to obtain a fourth vector representation. This includes: converting the format of the first scalar filtering condition to obtain a first expression; performing identifier processing on the first expression to obtain at least one second identifier; and processing the at least one second identifier to obtain the fourth vector representation. This allows the scalar filtering condition to be converted into a vector representation, facilitating subsequent integration with the vector representation of the material content data.

[0011] In one possible implementation, the weights of the third and fourth vector representations are different. This allows for a weighted merging of the two, improving the fusion effect and avoiding the performance degradation that would result from direct merging.

[0012] In one possible implementation, after loading the first vector representation into the vector index, the method further includes: receiving a retrieval request from a client, the retrieval request including user features; vectorizing the user features to obtain a user vector representation; performing a retrieval in the vector index based on the user vector representation to obtain at least one vector representation corresponding to the user vector representation; and returning the material content data represented by at least one vector representation to the client, or returning an identifier of at least one vector representation to the client. In this way, a single retrieval can simultaneously perform query filtering on material content data and scalar filtering conditions, resolving the inconsistency between vector and scalar retrieval targets, reducing losses before and after scalar filtering, and improving retrieval accuracy and performance.

[0013] In one possible implementation, the retrieval request also includes a second scalar filtering condition; the user features are vectorized to obtain a user vector representation, including: vectorizing the user features and the second scalar filtering condition using a neural network model to obtain a user vector representation, where the user vector representation is used to characterize the user features and the second scalar filtering condition. In this way, the inconsistency between vector and scalar retrieval targets can be resolved during the online retrieval phase by using a unified input standard, reducing the loss before and after scalar filtering, and improving retrieval accuracy. Simultaneously, the number and complexity of scalar filtering conditions can be disregarded during the online retrieval phase, thus solving the problem of increased overall latency and complexity in online retrieval and improving retrieval performance.

[0014] In one possible implementation, the retrieval request also includes: filtering strength; and before performing the retrieval in the vector index based on the user vector representation, it also includes: updating the user vector representation based on the filtering strength. In this way, the strength of scalar filtering can be controlled using the filtering strength, achieving scalar soft filtering.

[0015] In one possible implementation, the structure of the vector index includes nodes and edges, where nodes represent cluster centers and edges represent the affiliation relationships between nodes.

[0016] Secondly, this application provides a data processing method, comprising: receiving a retrieval request from a client, the retrieval request including user features and scalar filtering conditions; vectorizing the user features and scalar filtering conditions using a neural network model to obtain a user vector representation, wherein the user vector representation is used to characterize the user features and scalar filtering conditions; performing a retrieval among multiple material vector representations based on the user vector representation to obtain at least one material vector representation corresponding to the user vector representation, wherein each material vector representation is used to characterize the material content data and scalar filtering conditions of a material; and returning the material content data represented by at least one material vector representation to the client, or returning an identifier of at least one vector representation to the client.

[0017] Thirdly, this application provides a data processing apparatus comprising: an acquisition module and a processing module. The acquisition module is configured to acquire first material content data and a first scalar filtering condition related to the first material content data. The processing module is configured to perform vectorization processing on the first material content data and the first scalar filtering condition using a neural network model to obtain a first vector representation, wherein the first vector representation is used to characterize the first material content data and the first scalar filtering condition. The processing module is further configured to load the first vector representation into a vector index.

[0018] In one possible implementation, when the processing module performs vectorization processing on the first material content data and the first scalar filtering conditions through a neural network model to obtain a first vector representation, it specifically performs the following: vectorization processing on the first material content data to obtain a third vector representation; vectorization processing on the first scalar filtering conditions to obtain a fourth vector representation; and interactive learning on the third and fourth vector representations to obtain the first vector representation.

[0019] In one possible implementation, when the processing module performs vectorization processing on the first material content data to obtain a third vector representation, it specifically performs: identifier processing on the first material content data to obtain at least one first identifier; and performs interactive learning on the at least one first identifier to obtain a third vector representation.

[0020] In one possible implementation, when the processing module performs vectorization processing on the first scalar filtering condition to obtain the fourth vector representation, it specifically performs: format conversion on the first scalar filtering condition to obtain a first expression; performs identifier processing on the first expression to obtain at least one second identifier; and processes the at least one second identifier to obtain the fourth vector representation.

[0021] In one possible implementation, the weights of the third vector representation and the fourth vector representation are different in the fourth vector representation.

[0022] In one possible implementation, after loading the first vector representation into the vector index, the processing module is further configured to: receive a retrieval request from a client, the retrieval request including user features; perform vectorization processing on the user features to obtain a user vector representation; perform a retrieval in the vector index based on the user vector representation to obtain at least one vector representation corresponding to the user vector representation; and return to the client the material content data in the material represented by at least one vector representation, or return to the client the identifier of at least one vector representation.

[0023] In one possible implementation, the retrieval request also includes a second scalar filtering condition. In this case, when the processing module performs vectorization processing on the user features to obtain a user vector representation, it specifically performs vectorization processing on the user features and the second scalar filtering condition using a neural network model to obtain a user vector representation, wherein the user vector representation incorporates both the user features and the second scalar filtering condition.

[0024] In one possible implementation, the retrieval request also includes: filtering strength. In this case, before performing the retrieval in the vector index based on the user vector representation, the processing module also updates the user vector representation based on the filtering strength.

[0025] In one possible implementation, the structure of the vector index includes nodes and edges, where nodes represent cluster centers and edges represent the affiliation relationships between nodes.

[0026] Fourthly, this application provides a data processing apparatus comprising an acquisition module and a processing module. The acquisition module is configured to receive a retrieval request from a client, the retrieval request including user features and scalar filtering conditions. The processing module is configured to vectorize the user features and scalar filtering conditions using a neural network model to obtain a user vector representation, wherein the user vector representation is used to characterize the user features and scalar filtering conditions. The processing module is further configured to perform a retrieval among multiple material vector representations based on the user vector representation to obtain at least one material vector representation corresponding to the user vector representation, wherein each material vector representation integrates material content data of a material and scalar filtering conditions, and each material vector representation is used to characterize the material content data of a material. The processing module is further configured to return the material content data represented by at least one material vector representation to the client, or return an identifier of at least one vector representation to the client.

[0027] Fifthly, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect, or performs the method described in the second aspect or any possible implementation of the second aspect.

[0028] Sixthly, this application provides a computer-readable storage medium including computer program instructions. When executed by a computing device, the computing device performs the method described in the first aspect or any possible implementation thereof; or, when executed by a cluster of computing devices, the cluster of computing devices performs the method described in the first aspect or any possible implementation thereof, or performs the method described in the second aspect or any possible implementation thereof. Exemplarily, the cluster of computing devices may include one or more computing devices.

[0029] In a seventh aspect, this application provides a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in the first aspect or any possible implementation thereof, or to perform the method described in the second aspect or any possible implementation thereof. Exemplarily, the cluster of computing devices may include one or more computing devices.

[0030] It is understood that the beneficial effects of the second to seventh aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0031] Figure 1 is a schematic diagram of an application scenario provided by an embodiment of this application;

[0032] Figure 2 is a schematic diagram of a vector retrieval architecture with scalar filtering conditions provided in an embodiment of this application;

[0033] Figure 3 is a schematic diagram of another vector retrieval architecture with scalar filtering conditions provided in an embodiment of this application;

[0034] Figure 4 is a schematic diagram of the structure of a vector retrieval system with scalar filtering conditions provided in an embodiment of this application;

[0035] Figure 5 is a schematic diagram of a process for processing materials in a material warehouse during an offline stage, according to an embodiment of this application.

[0036] Figure 6 is a schematic diagram of a process for processing online search requests in the online stage according to an embodiment of this application;

[0037] Figure 7 is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0038] Figure 8 is a schematic diagram of the steps to obtain a vector representation that integrates scalar filtering conditions and material content data according to an embodiment of this application;

[0039] Figure 9 is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0040] Figure 10 is a schematic diagram of a data retrieval process provided in an embodiment of this application;

[0041] Figure 11 is a schematic diagram comparing search results provided in an embodiment of this application;

[0042] Figure 12 is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0043] Figure 13 is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0044] Figure 14 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0045] Figure 15 is a schematic diagram of another computing device cluster provided in an embodiment of this application. Detailed Implementation

[0046] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0047] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0048] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0049] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0050] First, the relevant technical terms used in this application will be introduced.

[0051] (1) Unstructured data

[0052] Unstructured data refers to unfiltered information without a fixed organizational structure. Common examples include web page logs, XML, JSON, text documents, images, video, and audio files.

[0053] (2) Structured data

[0054] Structured data refers to data that uses a standardized format, has a well-defined structure, conforms to a data model, follows a persistent order, and is easily accessible to humans and programs. This data type is typically stored in a database.

[0055] (3) Vector

[0056] A vector is a mathematical concept used to refer to a set of numerical values ​​of a specific length, which are treated as a whole for related calculations. A vector is an abstract representation of unstructured data, an ordered array of n numbers, i.e., an n-dimensional vector.

[0057] (4) Vector indexing

[0058] Vector indexes are a data structure needed for vector retrieval, which can improve the performance of vector approximate nearest neighbor (ANN) retrieval.

[0059] (5) Index building

[0060] Index building involves obtaining feature vectors from various types of unstructured data using inference representation techniques and writing them into the engine to complete the index building process.

[0061] (6) Vector Recall / Vector Retrieval

[0062] Vector recall / vector retrieval refers to using the features of an online request to transform it into a query vector, and then retrieving vector data from a vector index that is similar to the query vector. In this context, raw, unstructured data with similar meanings are close in distance within the vector space.

[0063] (7) Materials

[0064] Creative materials refer to various resources and elements designed to attract potential customers, increase brand awareness, boost website traffic, or promote sales. Materials can take the form of images, text, web pages, videos, or animations. They can be applied to search, recommendation, or advertising scenarios to enhance user engagement and conversion rates. For example, materials can be placed on a website or within web pages.

[0065] Materials can primarily consist of content data and scalar filtering conditions. Content data may include one or more of the following: title, tags, the smallest granular identifier of the content, promotional purpose, promotional application name, creative, copy, images, and videos. Scalar filtering conditions refer to the filtering expressions that must be met during material retrieval; these can be combinations of multiple filtering expressions. Scalar filtering conditions ensure that the content data accurately reaches the intended target audience.

[0066] (8) Pre-filter

[0067] Pre-filtering refers to scalar filtering performed before vector retrieval.

[0068] (9) Post-filter

[0069] Post-filtering refers to scalar filtering performed after vector retrieval.

[0070] (10) Vector retrieval with scalar filtering

[0071] Vector retrieval with scalar filtering refers to finding the vector x in the material library that is closest to q while meeting the filtering condition f, given a query vector q and a scalar filtering condition f.

[0072] (11) Vector Database

[0073] A vector database is a low-level library used to store vector data.

[0074] (12) Inverted index

[0075] An inverted index is an index storage format that maps words appearing in a document to a list of documents containing those words.

[0076] The technical solution provided in this application will be described below.

[0077] For example, Figure 1 illustrates an application scenario provided by an embodiment of this application. As shown in Figure 1, it demonstrates a recommendation advertising service system implemented using a vector retrieval engine. In Figure 1, the materials provided by advertisers can be pre-built into an index. Then, during ad delivery, user features are converted into user vector representations, and retrieval is performed in the pre-built index in combination with corresponding scalar filtering conditions, and the retrieved results are delivered to the user. The scenario shown in Figure 1 can be extended to search promotion (search, recommendation, advertising) services. Online retrieval requests from clients can include user features (such as user queries, user profiles, and contextual information viewed by the user). Search / advertising / recommendation scenarios involve multi-dimensional filtering conditions, such as scalar filtering conditions like targeting constraints and keyword matching.

[0078] For example, as shown in Table 1, consider the following advertising scenarios: Advertiser A advertises on "App 1," expecting their target audience to reside in Beijing, Shenzhen, or Shanghai, have "App 2" or "App 3" installed on their phones, be male, be between 20 and 30 years old, and have previously or currently searched for the keyword "travel." Advertiser B advertises on "App 4," expecting their target audience to reside in Beijing or Shenzhen, have "App 1" or "App 3" installed on their phones, with no restrictions on gender or age, and have previously or currently searched for the keyword "travel." The terms "can be bypassed, weak targeting" in Table 1 indicate that these conditions are not necessarily required.

[0079] Table 1

[0080] For user A, who has the following personal scalar attributes: male, residing in Beijing, has installed app 2, and has searched for "travel", their ad set is empty. This is because they do not meet advertiser A's requirement of "gender-reverse targeting: male" or advertiser B's requirement of "installed app list-targeting: {app 1, app 3}". For user B: female, 25 years old, residing in Shenzhen, has installed apps 2 and 3, and has searched for "travel", their ad set is {app 1, app 4}.

[0081] As can be seen from the above description, multi-dimensional filtering conditions exist in search / recommendation / advertising scenarios, such as scalar filtering conditions like targeting constraints and keyword matching. Quickly retrieving vector similarity results that meet scalar filtering conditions is a challenge in the industry.

[0082] Building upon this challenge, this application provides a vector retrieval architecture with scalar filtering conditions. As shown in Figure 2, this vector retrieval architecture can be divided into an offline part and an online part. In the offline part, data in the materials can be processed to create indexes for scalar filtering conditions and vector indexes for the material content data, which are then stored separately. In the online part, user queries can be encoded into vectors, and these vectors, along with the corresponding scalar filtering conditions, are input into the system for vector retrieval and scalar filtering respectively, ultimately returning the top N data points that meet the requirements. As can be seen from the architecture shown in Figure 2, the indexes for scalar filtering conditions and the vector indexes for the material content data are stored separately. Therefore, an additional external component is needed to store the indexes for scalar filtering conditions in the system. Furthermore, scalar filtering needs to be performed separately during online retrieval, which significantly increases the overall latency and complexity of online retrieval as the amount of scalar filtering data and the complexity of scalar conditions increase; simultaneously, the offline index building time and memory usage will also gradually increase. Furthermore, since there is no interaction between scalar filtering and vector retrieval, the retrieved nearest neighbor results contain a large number of materials that do not meet the scalar filtering conditions. These materials are pre-filtered / post-filtered in the ANN stage, which adds unnecessary storage and computation to the vector engine, resulting in no search results or insufficient recall, and wasting bandwidth.

[0083] In view of this, this application proposes another vector retrieval system with scalar filtering conditions. This system can transform material content data and scalar filtering conditions into a vector representation and construct a vector index. Furthermore, it can utilize the vector dimension to control and learn the clustering information of the scalar filtering conditions, performing vector retrieval by partition, thus achieving high-performance vector retrieval with complex scalar soft filtering functions under large data volumes. This eliminates the need for an additional external component to store the index of scalar filtering conditions, achieving a unified architecture and significantly improving the retrieval performance and effectiveness of vector retrieval with scalar filtering conditions.

[0084] For example, Figure 3 illustrates a schematic diagram of the technical concept of vector retrieval with scalar filtering conditions provided by an embodiment of this application. As shown in Figure 3, this technical concept is also divided into an offline part and an online part. In the offline part, a vector generation module that integrates scalar filtering can fuse scalar filtering conditions and material content data together to generate a vector representation, realizing the interaction between scalars and vectors. Then, a vector index structure can be constructed, and the corresponding vector representations can be loaded into the vector index for storage. Thus, scalar partitioning and filtering can be automatically realized by utilizing the clustering information of vector space distribution. Since the vector representation generated by the vector generation module that integrates scalar filtering conditions and material content data (i.e., the vector representation can represent scalar filtering conditions and material content data), a vector representation in the stored vector index can be associated with a material. A material can include material content data and at least one scalar filtering condition. In this way, it is no longer necessary to store the index of scalar filtering conditions separately, and no additional external components are needed, thereby saving disk and memory storage space for the index and improving the latency of offline index construction. Furthermore, since scalar filtering conditions do not require additional storage of inverted indexes and do not need to consider the complexity and number of scalars, storage resource consumption can be reduced, further expanding the retrieval scale. In other words, by controlling the complexity of scalars and the fineness of filtering in the vector dimension, the retrieval system can support large-scale complex scalar filtering conditions and further expand the retrieval scale.

[0085] Similarly, in the online component, a query generation module that integrates scalar filtering can vectorize user features (such as one or more of the user's query, user profile, and browsing context information) and corresponding scalar filtering conditions to merge them into a single vector representation. Then, an ANN retrieval method can be used to search within a vector index with scalar soft filtering capabilities, ultimately returning the top N data points that meet the requirements. For example, user features can refer to data that characterizes user profiles and / or behaviors, enabling personalized recommendations, search result optimization, and targeted advertising. Since user features and scalar filtering conditions are merged and transformed into a single vector representation during retrieval, a single search can simultaneously filter both content data and scalar filtering conditions. This unified input standard (i.e., both scalar filtering conditions and content data are input in vector representation) resolves the inconsistency between vector and scalar retrieval targets, reduces losses before and after scalar filtering, and improves retrieval accuracy. Meanwhile, by using a unified input standard, the number and complexity of scalar filtering conditions do not need to be considered during the retrieval stage, thus solving the problem of increased latency and complexity in online retrieval and improving retrieval performance.

[0086] The following describes the specific implementation process of the above technical concept.

[0087] For example, Figure 4 illustrates a schematic diagram of the architecture of a vector retrieval system with scalar filtering conditions provided in an embodiment of this application. As shown in Figure 4, the vector retrieval system 400 can provide offline and online services. The offline service is an offline index building service for a vector engine, mainly used to handle the construction of "vector indexes with scalar soft filtering functions," building vector indexes based on vector implicit partitioning (i.e., implicit scalar partitioning through vectors) soft filtering. The online service is a vector online retrieval service component for a search promotion (search, recommendation, advertising) platform, mainly used to handle online search / recommendation / advertising requests. It can generate vector representations that incorporate scalar filtering conditions, load vector indexes with scalar soft filtering functions, and complete vector retrieval with scalar filtering functions.

[0088] The vector retrieval system 400 can be configured with a material library 410, a vector generation module 420 integrating scalar filtering, an index building module 430, and a vector index 440 with scalar soft filtering functionality, so that the vector retrieval system 400 can provide offline services. The material library 410 is mainly used to store materials. Each material may include: material content data and at least one scalar filtering condition. For example, referring to Figure 5, Figure 5(A) shows a material provided by advertiser A, which contains the scalar filtering conditions and material content data described in Figure 5(B). Figure 5(B) is a data representation after normalizing the scalar filtering conditions and material content data in the material, which facilitates subsequent unified vectorization of both, converting them into a single vector representation. In some embodiments, when normalizing the scalar filtering condition material content data, the scalar filtering conditions can be converted into a fixed format (e.g., Boolean expression or JSON format). For example, the scalar filtering conditions can be converted into Boolean expressions, transforming the filtering logic into operator descriptions such as In, AND, OR, NOT, etc. When standardizing material content data, the data can be identified (IDed), ultimately outputting a series of IDs. These IDs can, but are not limited to, forming an ID list. For example, when an element in the material content data is already an ID (e.g., the creative ID and task ID in Figure 5(A)), it can be mapped to an ID map to obtain the corresponding ID. Conversely, when an element is not an ID (e.g., the promotional purpose in Figure 5(A)), it can be segmented into words, and the segmented data can be mapped to the ID map to obtain the corresponding ID. This allows different elements to be mapped to a unified space. It should be understood that material content data can also be standardized using methods (such as enumeration), depending on the specific circumstances, and is not limited here.

[0089] The vector generation module 420 for scalar filtering is mainly used to vectorize scalar filtering conditions and material content data (i.e., scalar filtering conditions and material content data within a material) that have a relationship. This allows the scalar filtering conditions to be vectorized and, together with the material content data, generated into a vector representation, thus achieving the fusion of scalar filtering conditions and material content data. Specifically, the vector generation module 420 generates a vector representation for each material it processes. Each vector representation generated by the scalar filtering module 420 represents the material content data of a material, as well as the scalar filtering conditions for that material. Within the scalar filtering module 420, scalar filtering conditions and material content data with a relationship can interact in various ways, including intra-scalar interaction, intra-material interaction, and scalar-vector interaction. This allows them to learn from each other, facilitating the grouping of similar ideas and scalars together during index construction. In some embodiments, continuing to refer to Figure 5, as shown in Figure 5(C), the vector generation module 420 for scalar filtering may include: a binarization layer 421, an identifier layer 422, a logical layer 423, a data encoder 424, and a fusion block 425. The binarization layer 421 is mainly used to binarize the range-type conditions in the scalar filtering conditions to ID the range-type conditions. The identifier layer 422 is mainly used to process the match-type conditions in the scalar filtering conditions. For example, match-type conditions can be segmented into words, and the segmented data can be mapped to an identifier map to ID the match-type conditions. After both range-type and match-type conditions are ID-ified, they can be merged, for example, by concatenation, to obtain an ID list of the scalar filtering conditions. The logic layer 423 is mainly used to process the ID list of scalar filtering conditions to perform scalar internal interaction, enabling the data in the scalar filtering conditions to learn each other's information, thereby obtaining the vector representation Zr of the scalar filtering conditions. For example, the logic layer 423 may include a conjunction operator network and a disjunction operator network. The conjunction operator network can be used for interaction between different scalars, and the disjunction operator network can be used for interaction between the same scalars. In Figure 5(C), hj is the hidden layer; when j=1, it is the input layer; Wtj is the network parameter matrix.The data encoder 424 is mainly used to process the ID list of material content data to enable internal interaction of the material content data, allowing the data in the material content data to learn each other's information, thereby obtaining the vector representation Zd of the material content data. For example, when merging vector representations Zr and Zd, they can be merged using a weighted method to improve the fusion effect and avoid the performance degradation caused by direct merging. In Figure 5(C), α represents the vector representation weight of the scalar filtering condition, and 1-α represents the vector representation weight of the material content data. α can be learned or is a hyperparameter; this is not limited here. The fusion module 425 is mainly used to process the merged vector representations Zr and Zd to enable interaction between the scalar filtering condition and the material content data, allowing them to learn each other's information, thereby obtaining a vector representation fused with the scalar filtering condition and the material content data. In this embodiment, both the logic layer 423 and the fusion module 425 can be, but are not limited to, multilayer perceptrons. Furthermore, using a logic layer 423 that supports both conjunction and disjunction paradigms can also make the results more interpretable. In some embodiments, during the training process of the vector generation module 420, the objective function (i.e., loss function) L can be used to control the loss ratio of scalar filtering conditions and material content data to perform deep representation learning, so as to simultaneously meet business objectives and corresponding rules and improve the delivery effect. In this embodiment, after the vector generation module 420 processes the materials in the material library, it can obtain the vector representation of the corresponding materials. Each vector representation incorporates the corresponding material content data and scalar filtering conditions. For example, Figure 5(D) shows a series of vector representations.

[0090] The index building module 430 primarily uses industry-standard vector algorithms (such as inverted file (IVF) classes, graphs, etc.) to process the vector representation generated by the vector generation module 420 to construct a vector index. Regardless of the vector algorithm used, the index building module 430's principle for constructing the vector index is to aggregate similar vectors together, facilitating quick retrieval of regions similar to the target vector and traversing within those regions. For example, the index building module 430 can process the vector representation output by the vector generation module 420 using clustering algorithms such as K-Nearest Neighbors (KNN) to learn the structure of the corresponding vector index. For example, the structure of the vector index can include nodes and edges. Nodes can indicate cluster centers, and edges can indicate the relationships between nodes. After obtaining the structure of the vector index, the vector representation output by the vector generation module 420 can be loaded into the corresponding nodes (also called "buckets") of the vector index. Within each bucket, the distance between the bucket's center and its internal vector representation is less than the distance between the internal vector representation and the centers of other buckets. For example, assuming there are 100,000 items in the material library, 100,000 vector representations can be obtained. Then, the structure of the vector index can be learned from all or part of these 100,000 vector representations to obtain the structure of the vector index. For example, the index structure can be a tree structure, where the first-level index includes the index values ​​of multiple cluster centers in multiple clusters, and the second-level index includes the index values ​​of each cluster. After obtaining the structure of the vector index, the 100,000 vector representations can be loaded into the corresponding buckets in the vector index, meaning each vector representation is assigned to a corresponding bucket. For instance, the index building module 430 can first parse the vector representations output by the vector generation module 420; then, it can train an index structure model of type IVF or similar using the parsed data; finally, it can construct the vector index using the trained index structure model. It should be understood that, because the vector representation output by the vector generation module 420 incorporates interactions between the scalar filtering conditions, the material content data, and between the two, the vector representation output by the vector generation module 420 integrates the features of both the scalar filtering conditions and the material content data. This allows for the grouping of similar types of creatives and scalars together during index construction. For example, as shown in Figure 5(E), only ads with similar scalar filtering conditions and behaviors will be grouped into the same bucket.

[0091] The vector index 440 with scalar soft filtering function is mainly used to store the index structure built by the index building module 430. It can be understood as an index data storage module.

[0092] Additionally, the vector retrieval system 400 can be configured with an interface 450, a query generation module 460 incorporating scalar filtering, and a vector retrieval module 470, so that the vector retrieval system 400 can provide online services. The interface 450 is primarily used to receive and process online retrieval requests from clients. These online retrieval requests may include user characteristics. Of course, the online retrieval requests may also include scalar filtering conditions and / or filtering strengths. For example, please refer to Figure 6; Figure 6(A) shows an online retrieval request that includes user characteristics, scalar filtering conditions, and filtering strengths.

[0093] The query generation module 460, which integrates scalar filtering, is primarily used to transform online retrieval requests into vector representations. This module may include a feature processing module 361, a scalar processing module 362, and a representation model (i.e., an embedding model) 363. The feature processing module 361 preprocesses the data in the online retrieval request, such as deduplication and tokenization. The scalar processing module 362 primarily converts the scalar filtering conditions in the online retrieval request into formats, such as performing Boolean expression conversion and transforming the filtering logic into operator descriptions like IN, AND, OR, and NOT. Through the feature processing module 361 and the scalar processing module 362, the online retrieval request can be transformed into a standardized data representation. The embedding model 363 primarily transforms the data processed by the feature processing module 361 and the scalar processing module 362 into a vector representation. When the online search request includes a filtering intensity, the query generation module 460 can multiply its generated vector representation, which integrates user features and scalar filtering conditions, with the filtering intensity to obtain the desired user vector representation. For example, the working principle of the query generation module 460 is similar to that of the aforementioned vector generation module 420, as detailed above, and will not be repeated here. The vector retrieval module 470 is mainly used to complete the ANN retrieval and return the TOP-K results. For example, referring to Figure 6, as shown in Figure 6(B), when there are multiple online search requests, the query generation module 460 can process each online search request separately to obtain the user vector representation corresponding to each online search request. Then, as shown in Figure 6(C), an ANN retrieval is performed on each user vector representation. Finally, as shown in Figure 6(D), for each online search request, the Top-K recall results can be retrieved and returned to the client. As described in the online service phase, scalar filtering conditions and user characteristics are input into the query generation module to obtain a vector representation of the user that incorporates the scalar filtering conditions. Furthermore, during the vector retrieval scanning process, data that basically meets the scalar filtering conditions is found through vector retrieval, thus achieving controllability of scalar filtering. In this way, by uniformly converting user characteristics and scalar filtering conditions in the online retrieval request into vector representations, it is unnecessary to calculate the vector representation of material content data and parse the scalar filtering conditions separately in two modules, accelerating the initial retrieval preparation work and improving performance. Additionally, when the online retrieval request includes filtering intensity, the strength of scalar filtering can be controlled using the filtering intensity, achieving soft scalar filtering.

[0094] The following sections will introduce the specific implementation processes of offline and online services in the Vector Retrieval System 400.

[0095] (1) Specific implementation process of offline service

[0096] For example, Figure 7 shows a schematic flowchart of a data processing method provided in an embodiment of this application. As shown in Figure 7, it can be understood that this method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. For example, this method can be executed by a data processing device, which can be implemented by software and / or hardware, and can be configured in, but is not limited to, a computing device cluster containing at least one computing device; typically, it can be configured in a server. For ease of description, the following description will focus on a server as the execution entity. As shown in Figure 7, the data processing method may include the following steps:

[0097] S701. Obtain a first material from the material library. The first material includes: first material content data and a first scalar filter condition, wherein the first scalar filter condition is a filter expression condition that must be satisfied during the retrieval of the first material content data.

[0098] In this embodiment, the server can load materials from the material library. Of course, the server can also obtain the first material from other sources; this is not limited here. A material can include its material content data and its scalar filtering conditions. The material content data and scalar filtering conditions in a material are related. For example, a material can have at least one scalar filtering condition, meaning there is at least one first scalar filtering condition. Furthermore, the scalar filtering conditions in a material are filtering expression conditions that must be satisfied during the retrieval of the material content data in that material. For example, as shown in Figure 5(A), which illustrates a material provided by advertiser A; the specific scalar filtering conditions and material content data in this material can be shown in Figure 5(B).

[0099] S702. The first material content data and the first scalar filtering condition are vectorized using a neural network model to obtain a first vector representation, wherein the first vector representation is used to characterize the first material content data and the first scalar filtering condition.

[0100] In this embodiment, after obtaining the first material, the server can use a neural network model to uniformly vectorize the first material content data and the first scalar filtering conditions to obtain a first vector representation. That is, the first material content data and the first scalar filtering conditions can be input into the neural network model for processing to obtain the first vector representation. The first vector representation is used to characterize the first material content data and the first scalar filtering conditions. In this embodiment, the neural network model can fuse the first material content data and the first scalar filtering conditions together to form a unified vector representation (i.e., the first vector representation); however, the two are not simply concatenated together, but need to be mutually integrated, such as through interactive learning.

[0101] As one possible implementation, as shown in Figure 8, the process of obtaining the first vector may include: in S801, vectorizing the first material content data to obtain a third vector representation. Specifically, the first material content data can first be identified to convert it into normalized data, thereby obtaining at least one first identifier (this process can be referred to the relevant description in Figure 4 above, and will not be repeated here). Then, interactive learning is performed on at least one first identifier to obtain a third vector representation. In this way, interactive learning of data within the material content data can be realized, so that the data in the material content data can learn each other's information, improve the accuracy of the obtained vector representation, and facilitate the subsequent aggregation of similar data in the material content data. For example, the neural network model used in S702 may be, but is not limited to, the vector generation module 420 described in Figure 4 above. In this case, the third vector representation can be obtained by processing the first identifier through the data encoder 424. In S802, vectorizing the first scalar filtering condition to obtain a fourth vector representation. Specifically, the first scalar filtering condition can be format-converted, such as converting it into a Boolean logical expression, to obtain a first expression. Then, the first expression is further processed to obtain at least one second identifier (this process can be referred to in the relevant description in Figure 4 above, and will not be repeated here). Finally, interactive learning is performed on at least one second identifier to obtain a fourth vector representation. In this way, the scalars in the scalar filtering conditions can learn each other's information, improving the accuracy of the obtained vector representation and facilitating the subsequent aggregation of similar scalar data, reducing the loss after merging scalar filtering conditions. For example, the neural network model used in S702 can be, but is not limited to, the vector generation module 420 described in Figure 4 above. In this case, the fourth vector representation can be obtained by processing at least one second identifier through the logic layer 423. In S803, interactive learning is performed on the third and fourth vector representations to obtain a first vector representation. In this way, the material content data and the scalar filtering conditions can learn each other's information, thereby facilitating the aggregation of similar types of ideas and similar types of scalars when building the index. For example, the neural network model used in S702 can be, but is not limited to, the vector generation module 420 described in Figure 4 above. In this case, the first vector representation can be obtained by processing the third and fourth vector representations through the fusion module 425. In some embodiments, during the processing of the third and fourth vector representations by the neural network model, the weights of the third and fourth vector representations are different, thereby merging them in a weighted manner to improve the fusion effect and avoid the effect degradation caused by direct merging. In some embodiments, the third vector representation can be understood as the vector representation Zr in Figure 5(C), and the fourth vector representation can be understood as the vector representation Zr in Figure 5(C).

[0102] S703. Load the first vector representation into the vector index.

[0103] In this embodiment, after obtaining the first vector representation, the server can load the first vector representation into the vector index for storage. In some embodiments, the structure of the vector index may include nodes and edges. Nodes can be used to indicate cluster centers, and edges can be used to indicate the affiliation relationships between nodes. When loading the first vector representation into the vector index, the first vector representation can be matched with the nodes in the structure of the vector index according to the node hierarchy until the most similar node is matched, and the first vector representation is stored in that node. In some embodiments, the server can process each material in the material library to obtain the vector representation of each material. Then, the server can use industry-standard vector algorithms (such as IVF classes, graphs, etc.) to process all or a portion of the obtained vector representations to learn the structure of the vector index. After learning the structure of the vector index, the server can load the vector representation corresponding to each material into the vector index. The process of learning the structure of the vector index is detailed in the aforementioned description of the index construction module 430 in Figure 4, and will not be repeated here.

[0104] In this way, by uniformly converting material content data and scalar filtering conditions into vector representations and constructing a vector index, it is no longer necessary to store the scalar filtering condition index separately, nor is it necessary to consider the number and complexity of scalar filtering conditions. This eliminates the need for additional external components, saving disk and memory storage space for the index and improving the latency of offline index construction. Furthermore, with this index structure, during the online retrieval phase, by using a unified input standard (i.e., both scalar filtering conditions and material content data are input as vector representations), and through a single retrieval, it is possible to simultaneously query and filter both material content data and scalar filtering conditions. This solves the problem of inconsistency between vector and scalar retrieval targets, reduces losses before and after scalar filtering, and improves retrieval accuracy. Simultaneously, because of the unified input standard, the number and complexity of scalar filtering conditions do not need to be considered during the retrieval phase, thus solving the problem of increased latency and complexity in online retrieval and improving retrieval performance.

[0105] (2) Specific implementation process of online services

[0106] For example, Figure 9 shows a schematic flowchart of a data processing method provided in an embodiment of this application. As shown in Figure 9, it can be understood that this method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. For example, this method can be executed by a data processing device, which can be implemented by software and / or hardware, and can be configured in, but is not limited to, a computing device cluster containing at least one computing device; typically, it can be configured in a server. For ease of description, the following description will focus on a server as the execution entity. As shown in Figure 9, the data processing method may include the following steps:

[0107] S901. Receive a retrieval request from a client, the retrieval request including user characteristics.

[0108] In this embodiment, when a user uses the client, the client can send a search request to the server. This search request may include user characteristics, such as the user's query, user profile, and contextual information the user has viewed. Of course, the search request may also include scalar filtering conditions and / or filtering strength. This is not limited here. For example, user characteristics can refer to data that can characterize a user profile and / or behavior. Personalized recommendations, search result optimization, and targeted advertising can be achieved through user characteristics.

[0109] S902. Perform vectorization processing on user features to obtain user vector representations.

[0110] In this embodiment, the server can vectorize the user features in the retrieval request to obtain a user vector representation. For example, the user features can be converted into a vector representation using the query generation module 460 in Figure 4, or other neural network models can be used to convert the user features into a vector representation, etc. Furthermore, when the retrieval request also includes a scalar filtering condition (hereinafter referred to as the "second scalar filtering condition"), the server can vectorize the user features and the second scalar filtering condition using a neural network model (e.g., the query generation module 460 described in Figure 4) to obtain a user vector representation. In this case, the user vector representation can be used to characterize the user features and the second scalar filtering condition. Moreover, when the retrieval request includes a filtering strength, after obtaining the user vector representation, the server can update the user vector representation using the filtering strength, for example, by multiplying the two, so as to control the strength of the scalar filtering using the filtering strength, thereby achieving scalar soft filtering.

[0111] S903. Based on the user vector representation, perform a search in the vector index to obtain at least one vector representation corresponding to the user vector representation.

[0112] In this embodiment, the server can process the user vector representation and vector index based on algorithms such as similarity or clustering to retrieve at least one vector representation corresponding to the user vector representation from the vector index (which is constructed in the offline stage). A vector representation in the vector index can be used to characterize the material content data and scalar filtering conditions contained in a material. For example, the vector representation in the vector index can also be called a "material vector representation". Furthermore, the similarity between the retrieved vector representation and the user vector representation meets preset requirements. For example, as shown in Figure 10, during retrieval, in S1001, an IVF type index retrieval is first used to obtain a list of scanned buckets. In S1002, for each bucket to be scanned, each vector representation within the bucket is traversed. In S1003, algorithms such as cosine similarity are used to calculate the vector similarity between the user vector representation and the corresponding vector representation in the bucket. In S1004, it is determined whether the recall result needs to meet a similarity threshold, for example, the similarity must be greater than 0.8. If not, proceed to S1005; otherwise, proceed to S1006. In S1005, the K most similar vector representations are returned, representing the associated material content data; that is, the material content data represented by the K vector representations are returned. In S1006, vector representations with similarity less than a similarity threshold are filtered out. In S1007, the material content data associated with the vector representations that meet the similarity threshold condition are returned; that is, the material content data represented by the corresponding vector representations are returned. In this way, the desired results are retrieved from the index structure.

[0113] S904. Return to the client the material content data represented in at least one vector representation, or return to the client the identifier of at least one vector representation.

[0114] In this embodiment, after retrieving the required vector representation, the server can query the corresponding material from the material library using the identifier of the vector representation and return the material content data of the queried material to the client. Each vector representation can have a unique identifier, and each vector representation identifier can have a mapping relationship with a material. Alternatively, the server can directly return the identifier of the queried vector representation to the client, allowing the client to query the corresponding material content data itself; the specific method depends on the actual situation and is not limited here.

[0115] In this way, since a single search can simultaneously query and filter both material content data and scalar filtering conditions, the inconsistency between vector and scalar search targets is resolved, losses before and after scalar filtering are reduced, and search accuracy is improved. Furthermore, the number and complexity of scalar filtering conditions do not need to be considered during the search phase, thus addressing the issues of increased latency and complexity in online searches and improving search performance.

[0116] The effects of the solutions provided in the embodiments of this application will be described below.

[0117] As shown in Figure 11, comparing the retrieval method provided in this solution with that in related technologies, it can be seen that, as shown in Figure 11(A), when the number of indexes is 5 million, the queries per second (QPS) is 500, and the number of recalled results is 200, the online retrieval latency in related technologies increases significantly with the increase of the number of scalar filter conditions, while the online retrieval latency in this solution is independent of the number of scalar filter conditions. As shown in Figure 11(B), when the number of indexes is 5 million, the number of scalar filter conditions is 20, and the number of recalled results is 200, the online retrieval latency in related technologies increases significantly with the increase of the complexity of the scalar filter logic, while the online retrieval latency in this solution is independent of the complexity of the scalar filter logic.

[0118] As shown in Figure 11(C), in the scenarios of search advertising and display advertising, the memory usage is high when using an external component to store scalar filtering conditions (i.e., storing them through an inverted index). However, this solution eliminates the storage of the inverted index (i.e., it does not require an additional external component to store the index of scalar filtering conditions), and the memory space usage is significantly reduced compared with the retrieval methods in related technologies.

[0119] As shown in Figure 11(D), among the 100 recalled data points, the results retrieved using this solution that satisfy both scalar filtering conditions A and B are higher than those retrieved using related technologies. In other words, the scalar condition hit rate of the offline recall results is also significantly improved, achieving the filtering function using vector retrieval, significantly reducing the loss before and after scalar filtering, and improving retrieval performance.

[0120] It is understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In addition, the various embodiments or technical features involved in the embodiments described above can be combined according to the actual situation, and the combined solution is still within the protection scope of this application.

[0121] Based on the methods in the above embodiments, this application also provides a data processing apparatus.

[0122] For example, Figure 12 shows a schematic diagram of a data processing device provided in an embodiment of this application. As shown in Figure 12, the data processing device 1200 includes an acquisition module 1201 and a processing module 1202. The acquisition module 1201 is used to acquire a first material, which includes first material content data and a first scalar filtering condition. The processing module 1202 is used to perform vectorization processing on the first material content data and the first scalar filtering condition using a neural network model to obtain a first vector representation, wherein the first vector representation is used to characterize the first material content data and the first scalar filtering condition. The processing module 1202 is also used to load the first vector representation into a vector index.

[0123] In some embodiments, when the processing module 1202 performs vectorization processing on the first material content data and the first scalar filtering conditions through a neural network model to obtain a first vector representation, it is specifically used to: perform vectorization processing on the first material content data to obtain a third vector representation; perform vectorization processing on the first scalar filtering conditions to obtain a fourth vector representation; and perform interactive learning on the third vector representation and the fourth vector representation to obtain the first vector representation.

[0124] In some embodiments, when the processing module 1202 performs vectorization processing on the first material content data to obtain a third vector representation, it is specifically used to: perform identifier processing on the first material content data to obtain at least one first identifier; and perform interactive learning on the at least one first identifier to obtain a third vector representation.

[0125] In some embodiments, when the processing module 1202 performs vectorization processing on the first scalar filtering condition to obtain a fourth vector representation, it is specifically used to: perform format conversion on the first scalar filtering condition to obtain a first expression; perform identifier processing on the first expression to obtain at least one second identifier; and process the at least one second identifier to obtain a fourth vector representation.

[0126] In some embodiments, the weight ratios of the third vector representation and the fourth vector representation are different.

[0127] In some embodiments, after loading the first vector representation into the vector index, the processing module 1202 is further configured to: receive a retrieval request from a client, the retrieval request including user features; perform vectorization processing on the user features to obtain a user vector representation; perform a retrieval in the vector index based on the user vector representation to obtain at least one vector representation corresponding to the user vector representation; return the material content data represented by at least one vector representation to the client, or return the identifier of at least one vector representation to the client.

[0128] In some embodiments, the retrieval request further includes a second scalar filtering condition. In this case, when the processing module 1202 performs vectorization processing on the user features to obtain a user vector representation, it specifically performs: vectorization processing on the user features and the second scalar filtering condition using a neural network model to obtain a user vector representation, wherein the user vector representation is used to characterize the user features and the second scalar filtering condition.

[0129] In some embodiments, the retrieval request further includes: filtering strength. At this time, before performing the retrieval in the vector index based on the user vector representation, the processing module 1202 is also used to: update the user vector representation based on the filtering strength.

[0130] In some embodiments, the structure of the vector index includes nodes and edges, where nodes represent cluster centers and edges represent the affiliation relationships between nodes.

[0131] In some embodiments, both the acquisition module 1201 and the processing module 1202 shown in FIG12 can be implemented in software or in hardware. For example, the implementation of the acquisition module 1201 will be described below. Similarly, the implementation of the processing module 1202 can refer to the implementation of the acquisition module 1201.

[0132] As an example of a software functional unit, module 1201 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, module 1201 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0133] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0134] As an example of a hardware functional unit, the acquisition module 1201 may include at least one computing device, such as a server. Alternatively, the acquisition module 1201 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0135] The multiple computing devices included in the acquisition module 1201 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the acquisition module 1201 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the acquisition module 1201 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0136] It should be noted that, in other embodiments, the acquisition module 1201 can be used to execute any step in the data processing method described in the above embodiments, and the processing module 1202 can be used to execute any step in the data processing method described in the above embodiments. Furthermore, the steps implemented by the acquisition module 1201 and the processing module 1202 can be specified as needed. By implementing different steps in the data processing method described in the above embodiments through the acquisition module 1201 and the processing module 1202, all the functions of the data processing device 1200 shown in FIG12 can be achieved.

[0137] This application also provides a computing device 1300. As shown in FIG13, the computing device 1300 includes: a bus 1302, a processor 1304, a memory 1306, and a communication interface 1308. The processor 1304, the memory 1306, and the communication interface 1308 communicate with each other via the bus 1302. The computing device 1300 may be a server or an electronic device. It should be understood that this application does not limit the number of processors and memories in the computing device 1300.

[0138] Bus 1302 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 13, but this does not imply that there is only one bus or one type of bus. Bus 1304 can include pathways for transmitting information between various components of computing device 1300 (e.g., memory 1306, processor 1304, communication interface 1308).

[0139] The processor 1304 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0140] The memory 1306 may include volatile memory, such as random access memory (RAM). The processor 1304 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0141] The memory 1306 stores executable program code, and the processor 1304 executes the executable program code to implement the functions of the acquisition module 1201 and the processing module 1202 shown in FIG. 12, thereby implementing the data processing method described in the above embodiments. That is, the memory 1306 stores instructions for executing the data processing method described in the above embodiments.

[0142] Alternatively, the memory 1306 may store executable code, which the processor 1304 executes to implement the functions of the data processing apparatus 1200 shown in FIG. 12, thereby implementing the data processing method described in the above embodiments. That is, the memory 1306 stores instructions for executing the data processing method described in the above embodiments.

[0143] The communication interface 1303 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1300 and other devices or communication networks.

[0144] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be an electronic device such as a desktop computer, a laptop computer, or a smartphone.

[0145] As shown in Figure 14, the computing device cluster includes at least one computing device 1300. The memory 1306 of one or more computing devices 1300 in the computing device cluster may store the same instructions for executing the data processing methods described in the above embodiments.

[0146] In some possible implementations, the memory 1306 of one or more computing devices 1300 in the computing device cluster may also store partial instructions for executing the data processing method described in the above embodiments. In other words, a combination of one or more computing devices 1300 can jointly execute instructions for executing the data processing method described in the above embodiments.

[0147] It should be noted that the memory 1306 in different computing devices 1300 within the computing device cluster can store different instructions, which are used to execute some of the functions of the data processing device 1200 shown in Figure 12. That is, the instructions stored in the memory 1306 of different computing devices 1300 can implement the functions of one or more modules in the acquisition module 1201 and the processing module 1202.

[0148] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 15 illustrates one possible implementation. As shown in Figure 15, two computing devices 1300A and 1300B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 1306 in computing device 1300A stores instructions for executing the functions of the acquisition module 1201. Simultaneously, the memory 1306 in computing device 1300B stores instructions for executing the functions of the processing module 1202.

[0149] It should be understood that the functions of computing device 1300A shown in Figure 15 can also be performed by multiple computing devices 1300. Similarly, the functions of computing device 1300B can also be performed by multiple computing devices 1300.

[0150] This application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similarly referred to the connection method of the computing device cluster described in Figures 14 and 15. The difference is that the memory 1306 of one or more computing devices 1300 in this computing device cluster can store the same instructions for executing the methods in the above embodiments.

[0151] In some possible implementations, the memory 1306 of one or more computing devices 1300 in the computing device cluster may also store partial instructions for executing the aforementioned data processing method. In other words, a combination of one or more computing devices 1300 can jointly execute instructions for executing the aforementioned data processing method.

[0152] Based on the methods in the above embodiments, this application provides a computer-readable storage medium including computer program instructions. When the computer program instructions are executed by a computing device, the computing device performs the methods in the above embodiments; or, when the computer program instructions are executed by a cluster of computing devices, the cluster of computing devices performs the methods in the above embodiments. Exemplarily, the computer-readable storage medium can be any available medium that the computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0153] Based on the methods in the above embodiments, this application provides a computer program product containing instructions that, when executed by a computing device, cause the computing device to perform the methods in the above embodiments, or, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the methods in the above embodiments.

[0154] Based on the methods in the above embodiments, this application also provides a data processing apparatus, including: an acquisition module and a processing module. The acquisition module is used to receive a retrieval request from a client, the retrieval request including user features and scalar filtering conditions. The processing module is used to vectorize the user features and scalar filtering conditions using a neural network model to obtain a user vector representation, wherein the user vector representation is used to characterize the user features and scalar filtering conditions. The processing module is further used to perform a retrieval among multiple material vector representations based on the user vector representation to obtain at least one material vector representation corresponding to the user vector representation, wherein each material vector representation integrates material content data and scalar filtering conditions for one material, and each material vector representation is used to characterize the material content data of one material. The processing module is further used to return the material content data represented by at least one material vector representation to the client, or to return an identifier of at least one vector representation to the client.

[0155] It should be understood that the above-described device is used to execute the methods in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above methods. The working process of the device can be referred to the corresponding process in the above methods, and will not be repeated here.

[0156] It is understood that the processor in the embodiments of this application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor may be a microprocessor or any conventional processor.

[0157] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0158] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, cloud computing platform, or data center to another website, computer, cloud computing platform, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a cloud computing platform or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0159] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, The method includes: Obtain the first material content data and the first scalar filter condition related to the first material content data; The first material content data and the first scalar filtering condition are vectorized using a neural network model to obtain a first vector representation, wherein the first vector representation is used to characterize the first material content data and the first scalar filtering condition. Load the first vector representation into the vector index.

2. The method according to claim 1, characterized in that, The step of vectorizing the first material content data and the first scalar filtering conditions using a neural network model to obtain a first vector representation includes: The first material content data is vectorized to obtain a third vector representation; The first scalar filtering condition is vectorized to obtain a fourth vector representation; The third vector representation and the fourth vector representation are interactively learned to obtain the first vector representation.

3. The method according to claim 2, characterized in that, The vectorization process of the first material content data to obtain a third vector representation includes: The first material content data is tagged to obtain at least one first identifier; Interactive learning is performed on the at least one first identifier to obtain the third vector representation.

4. The method according to claim 2 or 3, characterized in that, The vectorization of the first scalar filtering condition to obtain a fourth vector representation includes: The first scalar filtering condition is formatted to obtain the first expression; The first expression is processed to obtain at least one second identifier; The at least one second identifier is processed to obtain the fourth vector representation.

5. The method according to any one of claims 2-4, characterized in that, The weights of the third vector representation and the fourth vector representation are different.

6. The method according to any one of claims 1-5, characterized in that, After loading the first vector representation into the vector index, the process further includes: Receive a retrieval request from a client, the retrieval request including user characteristics; The user features are vectorized to obtain a user vector representation; Based on the user vector representation, a search is performed in the vector index to obtain at least one vector representation corresponding to the user vector representation; Return the material content data represented by the at least one vector representation to the client, or return the identifier of the at least one vector representation to the client.

7. The method according to claim 6, characterized in that, The search request also includes a second scalar filtering condition; The process of vectorizing the user features to obtain a user vector representation includes: The user features and the second scalar filtering condition are vectorized using the neural network model to obtain a user vector representation, wherein the user vector representation is used to characterize the user features and the second scalar filtering condition.

8. The method according to claim 6 or 7, characterized in that, The retrieval request also includes: filtering strength; Before performing a retrieval in the vector index based on the user vector representation, the method further includes: The fourth vector representation is updated based on the filtering intensity.

9. The method according to any one of claims 1-8, characterized in that, The structure of the vector index includes nodes and edges, where nodes represent cluster centers and edges represent the affiliation relationships between nodes.

10. A data processing method, characterized in that, The method includes: Receive a search request from a client, the search request including user characteristics and scalar filtering conditions; The user features and the scalar filtering conditions are vectorized using a neural network model to obtain a user vector representation, wherein the user vector representation incorporates the user features and the scalar filtering conditions. Based on the user vector representation, a search is performed among multiple material vector representations to obtain at least one material vector representation corresponding to the user vector representation, wherein each material vector representation is used to characterize the material content data and scalar filtering conditions of a material; Return the material content data represented by the at least one material vector representation to the client, or return the identifier of the at least one vector representation to the client.

11. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire first material content data and first scalar filtering conditions related to the first material content data; The processing module is used to perform vectorization processing on the first material content data and the first scalar filtering condition through a neural network model to obtain a first vector representation, wherein the first vector representation is used to characterize the first material content data and the first scalar filtering condition. The processing module is further configured to load the first vector representation into the vector index.

12. The apparatus according to claim 11, characterized in that, When the processing module performs vectorization processing on the first material content data and the first scalar filtering conditions through a neural network model to obtain a first vector representation, it is specifically used for: The first material content data is vectorized to obtain a third vector representation; The first scalar filtering condition is vectorized to obtain a fourth vector representation; The third vector representation and the fourth vector representation are interactively learned to obtain the first vector representation.

13. The apparatus according to claim 12, characterized in that, When the processing module performs vectorization processing on the first material content data to obtain a third vector representation, it is specifically used for: The first material content data is tagged to obtain at least one first identifier; Interactive learning is performed on the at least one first identifier to obtain the third vector representation.

14. The apparatus according to claim 12 or 13, characterized in that, When the processing module performs vectorization processing on the first scalar filtering condition to obtain the fourth vector representation, it is specifically used for: The first scalar filtering condition is formatted to obtain the first expression; The first expression is processed to obtain at least one second identifier; The at least one second identifier is processed to obtain the fourth vector representation.

15. The apparatus according to any one of claims 12-14, characterized in that, The weights of the third vector representation and the fourth vector representation are different.

16. The apparatus according to any one of claims 11-15, characterized in that, After loading the first vector representation into the vector index, the processing module is further configured to: Receive a retrieval request from a client, the retrieval request including user characteristics; The user features are vectorized to obtain a user vector representation; Based on the user vector representation, a search is performed in the vector index to obtain at least one vector representation corresponding to the user vector representation; Return the material content data in the material represented by the at least one vector representation to the client, or return the identifier of the at least one vector representation to the client.

17. The apparatus according to claim 16, characterized in that, The search request also includes a second scalar filtering condition; When the processing module performs vectorization processing on the user features to obtain a user vector representation, it is specifically used for: The user features and the second scalar filtering condition are vectorized using the neural network model to obtain a user vector representation, wherein the user vector representation is used to characterize the user features and the second scalar filtering condition.

18. The apparatus according to claim 16 or 17, characterized in that, The retrieval request also includes: filtering strength; Before performing a search in the vector index based on the user vector representation, the processing module is further configured to: The user vector representation is updated based on the filtering intensity.

19. The apparatus according to any one of claims 11-18, characterized in that, The structure of the vector index includes nodes and edges, where nodes represent cluster centers and edges represent the affiliation relationships between nodes.

20. A data processing apparatus, characterized in that, include: The acquisition module is used to receive a retrieval request from the client, the retrieval request including user characteristics and scalar filtering conditions; The processing module is used to vectorize the user features and the scalar filtering conditions through a neural network model to obtain a user vector representation, wherein the user vector representation is used to characterize the user features and the scalar filtering conditions. The processing module is further configured to perform a search among multiple material vector representations based on the user vector representation to obtain at least one material vector representation corresponding to the user vector representation, wherein each material vector representation integrates material content data of a material and scalar filtering conditions, and each material vector representation is used to represent material content data of a material. The processing module is further configured to return the material content data represented by the at least one material vector representation to the client, or to return the identifier of the at least one vector representation to the client.

21. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-9, or to perform the method as described in claim 10.

22. A computer-readable storage medium, characterized in that, The method includes computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any one of claims 1-9, or to perform the method as described in claim 10, wherein the cluster of computing devices includes at least one computing device.

23. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1-9, or to perform the method as described in claim 10, wherein the computing device cluster includes at least one computing device.

Images