Data processing method and device and computing device cluster

Abstract

The data processing method comprises the following steps: acquiring first material content data and a first scalar filtering condition related to the first material content data; vectorization processing is carried out on the first material content data and the first scalar filtering condition through a neural network model to obtain first vector representation, and the first vector representation is used for representing the first material content data and the first scalar filtering condition; and loading the first vector representation into a vector index. According to the method, the performance and effect of vector retrieval with scalar filtering conditions can be remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence (AI), and particularly relates to a data processing method and device and a computing device cluster. BACKGROUND

[0002] With the wide application of AI technology in data retrieval, the search has evolved from keyword-based retrieval to vector retrieval based on data similarity. Through AI technology, not only can structured data be converted into multi-dimensional vectors, but also unstructured data can be converted into multi-dimensional vectors. In the vector retrieval process, the distance between vectors is usually used to represent the similarity of the corresponding data. However, in addition to vector data, there are other scalar data information as filtering conditions in user query requests, which requires that in the retrieval process, the semantic similarity is captured while considering additional filtering conditions. For example, in the legal field, only relevant legal provisions may be searched from a certain specific label database, and in the retail industry, only a certain size of men's shoes may be searched, and so on. Such problems can be described as vector retrieval with scalar filtering conditions.

[0003] In related technologies, in order to complete the vector retrieval with scalar filtering conditions, the scalar filtering is usually performed first and then the vector retrieval (Pre-filter), or the vector retrieval is performed first and then the scalar filtering (Post-filter). However, these retrieval methods usually have the problem of poor retrieval performance and effect. SUMMARY

[0004] The present application provides a data processing method, device, computing device cluster, computer storage medium and computer product, which can significantly improve the performance and effect of vector retrieval with scalar filtering conditions.

[0005] In a first aspect, the present application provides a data processing method, comprising: obtaining first material content data and a first scalar filtering condition related to the first material content data; performing vectorization processing on the first material content data and the first scalar filtering condition through a neural network model to obtain a first vector representation, wherein the first vector representation is used to represent the first material content data and the first scalar filtering condition; and loading the first vector representation into a vector index.

[0006] In this way, by uniformly converting the material content data and the scalar filtering condition into a vector representation, and constructing a vector index, the index of the scalar filtering condition does not need to be stored separately, and the number and complexity of the scalar filtering condition do not need to be considered, so that additional external components are no longer needed, thereby saving the disk and memory storage space of the index, and improving the latency of offline index construction. At the same time, under such an index structure, by uniformly inputting a standard (i.e., the scalar filtering condition and the material content data are both input as a vector representation), and performing a search, the query filtering of the material content data and the scalar filtering condition can be realized at the same time, the inconsistency problem of the vector and the scalar search target is solved, the loss before and after the scalar filtering is reduced, and the search precision is improved. At the same time, due to the uniform input standard, the number and complexity of the scalar filtering condition do not need to be considered at the search stage, so that the problem of increasing the overall latency and complexity of online search is solved, and the search performance is improved.

[0007] In a possible implementation, the neural network model is used to perform vectorization processing on the first material content data and the first scalar filtering condition to obtain a first vector representation, including: performing vectorization processing on the first material content data to obtain a third vector representation; performing vectorization processing on the first scalar filtering condition to obtain a fourth vector representation; and performing interactive learning on the third vector representation and the fourth vector representation to obtain the first vector representation. In this way, the first material content data and the first scalar filtering condition can be fused together. At the same time, during the fusion process, the first material content data and the first scalar filtering condition can interact with each other, so that both of them can learn the knowledge of the other, thereby facilitating the aggregation of the same type of creativity and the same type of scalar during index construction.

[0008] In a possible implementation, the vectorization processing on the first material content data to obtain a third vector representation includes: performing identification processing on the first material content data to obtain at least one first identifier; and performing interactive learning on the at least one first identifier to obtain the third vector representation. In this way, the first material content data can be converted into a vector representation. At the same time, during the conversion process, the data in the first material content data can interact with each other, so that different data can learn the knowledge of the other, improve the accuracy of the obtained vector representation, and facilitate the subsequent aggregation of similar material content data.

[0009] In a possible implementation, the first scalar filtering condition is vectorized to obtain the fourth vector representation, including: performing format conversion on the first scalar filtering condition to obtain a first expression; performing identification processing on the first expression to obtain at least one second identifier; and performing processing on the at least one second identifier to obtain the fourth vector representation. In this way, the scalar filtering condition can be converted into a vector representation, thereby facilitating subsequent fusion with the vector representation of the material content data.

[0010] In a possible implementation, the third vector representation and the fourth vector representation have different weights. In this way, the two can be combined by weighting, thereby improving the fusion effect of the two and avoiding the effect decline caused by direct combination.

[0011] In a possible implementation, after the first vector representation is loaded into the vector index, the method further includes: receiving a retrieval request from a client, the retrieval request including a user feature; performing vectorization processing on the user feature to obtain a user vector representation; performing retrieval in the vector index based on the user vector representation to obtain at least one vector representation corresponding to the user vector representation; and returning, to the client, material content data represented by the at least one vector representation, or returning, to the client, an identifier of the at least one vector representation. In this way, the material content data and the scalar filtering condition can be simultaneously queried and filtered by one retrieval, thereby solving the inconsistency between the vector and the scalar retrieval target, reducing the loss before and after the scalar filtering, and improving the retrieval accuracy and performance.

[0012] In a possible implementation, the retrieval request further includes a second scalar filtering condition; and the performing vectorization processing on the user feature to obtain the user vector representation includes: performing vectorization processing on the user feature and the second scalar filtering condition by using a neural network model to obtain the user vector representation, where the user vector representation is used to represent the user feature and the second scalar filtering condition. In this way, the unified input standard can be used in the online retrieval stage, thereby solving the inconsistency between the vector and the scalar retrieval target, reducing the loss before and after the scalar filtering, and improving the retrieval accuracy. Meanwhile, the number and complexity of the scalar filtering conditions do not need to be considered in the online retrieval stage, thereby solving the problems of increased overall time delay and complexity in the online retrieval, and improving the retrieval performance.

[0013] In a possible implementation, the retrieval request further includes a filtering intensity; and before the performing retrieval in the vector index based on the user vector representation, the method further includes: updating the user vector representation based on the filtering intensity. In this way, the filtering intensity can be used to control the strength of the scalar filtering, thereby implementing scalar soft filtering.

[0014] In a possible implementation, the structure of the vector index comprises: nodes, which are used to represent cluster centers, and edges, which are used to represent the membership between nodes.

[0015] In a second aspect, the present application provides a data processing method, comprising: receiving a search request from a client, the search request comprising user features and scalar filtering conditions; performing vectorization processing on the user features and the scalar filtering conditions by using a neural network model to obtain a user vector representation, wherein the user vector representation is used to represent the user features and the scalar filtering conditions; performing search in a plurality of material vector representations based on the user vector representation to obtain at least one material vector representation corresponding to the user vector representation, wherein each material vector representation is used to represent material content data and scalar filtering conditions of a material; returning, to the client, material content data represented by the at least one material vector representation, or returning, to the client, an identifier of the at least one vector representation.

[0016] In a third aspect, the present application provides a data processing apparatus, comprising: an acquisition module and a processing module. The acquisition module is configured to acquire first material content data and first scalar filtering conditions related to the first material content data. The processing module is configured to perform vectorization processing on the first material content data and the first scalar filtering conditions by using a neural network model to obtain a first vector representation, wherein the first vector representation is used to represent the first material content data and the first scalar filtering conditions. The processing module is further configured to load the first vector representation into a vector index.

[0017] In a possible implementation, when the processing module performs vectorization processing on the first material content data and the first scalar filtering conditions by using the neural network model to obtain the first vector representation, the processing module is specifically configured to: perform vectorization processing on the first material content data to obtain a third vector representation; perform vectorization processing on the first scalar filtering conditions to obtain a fourth vector representation; and perform interactive learning on the third vector representation and the fourth vector representation to obtain the first vector representation.

[0018] In a possible implementation, when the processing module performs vectorization processing on the first material content data to obtain the third vector representation, the processing module is specifically configured to: perform identification processing on the first material content data to obtain at least one first identifier; and perform interactive learning on the at least one first identifier to obtain the third vector representation.

[0019] In a possible implementation, when the processing module performs vectorization processing on the first scalar filtering conditions to obtain the fourth vector representation, the processing module is specifically configured to: perform format conversion on the first scalar filtering conditions to obtain a first expression; perform identification processing on the first expression to obtain at least one second identifier; and perform processing on the at least one second identifier to obtain the fourth vector representation.

[0020] In a possible implementation, the third vector representation and the fourth vector representation have different weights.

[0021] In a possible implementation, after loading the first vector representation into the vector index, the processing module is further configured to: receive a retrieval request from the client, the retrieval request including a user feature; perform vectorization processing on the user feature to obtain a user vector representation; perform retrieval in the vector index based on the user vector representation to obtain at least one vector representation corresponding to the user vector representation; and return, to the client, material content data in materials represented by the at least one vector representation, or return, to the client, an identifier of the at least one vector representation.

[0022] In a possible implementation, the retrieval request further includes a second scalar filter condition. In this case, when performing vectorization processing on the user feature to obtain the user vector representation, the processing module is specifically configured to: perform vectorization processing on the user feature and the second scalar filter condition by using the neural network model to obtain the user vector representation, wherein the user vector representation fuses the user feature and the second scalar filter condition.

[0023] In a possible implementation, the retrieval request further includes a filter intensity. In this case, before performing retrieval in the vector index based on the user vector representation, the processing module is further configured to: update the user vector representation based on the filter intensity.

[0024] In a possible implementation, the structure of the vector index includes: nodes and edges, the nodes being used to represent cluster centers, and the edges being used to represent the ownership relationship between the nodes.

[0025] In a fourth aspect, the present application provides a data processing apparatus, including: an obtaining module and a processing module. The obtaining module is configured to receive a retrieval request from a client, the retrieval request including a user feature and a scalar filter condition. The processing module is configured to perform vectorization processing on the user feature and the scalar filter condition by using a neural network model to obtain a user vector representation, wherein the user vector representation is used to represent the user feature and the scalar filter condition. The processing module is further configured to perform retrieval in a plurality of material vector representations based on the user vector representation to obtain at least one material vector representation corresponding to the user vector representation, wherein each material vector representation fuses material content data of a material and the scalar filter condition, and each material vector representation is used to represent the material content data of the material. The processing module is further configured to return, to the client, material content data represented by the at least one material vector representation, or return, to the client, an identifier of the at least one vector representation.

[0026] Fifthly, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect, or performs the method described in the second aspect or any possible implementation of the second aspect.

[0027] Sixthly, this application provides a computer-readable storage medium including computer program instructions. When executed by a computing device, the computing device performs the method described in the first aspect or any possible implementation thereof; or, when executed by a cluster of computing devices, the cluster of computing devices performs the method described in the first aspect or any possible implementation thereof, or performs the method described in the second aspect or any possible implementation thereof. Exemplarily, the cluster of computing devices may include one or more computing devices.

[0028] In a seventh aspect, this application provides a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in the first aspect or any possible implementation thereof, or to perform the method described in the second aspect or any possible implementation thereof. Exemplarily, the cluster of computing devices may include one or more computing devices.

[0029] It is understood that the beneficial effects of the second to seventh aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0030] Figure 1 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0031] Figure 2 This is a schematic diagram of a vector retrieval architecture with scalar filtering conditions provided in an embodiment of this application;

[0032] Figure 3 This is a schematic diagram of another vector retrieval architecture with scalar filtering conditions provided in the embodiments of this application;

[0033] Figure 4 This is a schematic diagram of the structure of a vector retrieval system with scalar filtering conditions provided in an embodiment of this application;

[0034] Figure 5 This is a schematic diagram illustrating the offline processing of materials in a material warehouse, as provided in an embodiment of this application.

[0035] Figure 6 This is a schematic diagram illustrating a process for processing online search requests during the online phase, as provided in an embodiment of this application.

[0036] Figure 7 This is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0037] Figure 8 This is a schematic diagram illustrating the steps of obtaining a vector representation that integrates scalar filtering conditions and material content data, as provided in an embodiment of this application.

[0038] Figure 9 This is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0039] Figure 10 This is a schematic diagram illustrating a data retrieval process provided in an embodiment of this application;

[0040] Figure 11 This is a comparative diagram of search results provided in an embodiment of this application;

[0041] Figure 12 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0042] Figure 13 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0043] Figure 14 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0044] Figure 15 This is a schematic diagram of another computing device cluster structure provided in an embodiment of this application. Detailed Implementation

[0045] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0046] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0047] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0048] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0049] First, the relevant technical terms used in this application will be introduced.

[0050] (1) Unstructured data

[0051] Unstructured data refers to unfiltered information without a fixed organizational structure. Common examples include web page logs, XML, JSON, text documents, images, video, and audio files.

[0052] (2) Structured data

[0053] Structured data refers to data that uses a standardized format, has a well-defined structure, conforms to a data model, follows a persistent order, and is easily accessible to humans and programs. This data type is typically stored in a database.

[0054] (3) Vector

[0055] A vector is a mathematical concept used to refer to a set of numerical values ​​of a specific length, which are treated as a whole for related calculations. A vector is an abstract representation of unstructured data, an ordered array of n numbers, i.e., an n-dimensional vector.

[0056] (4) Vector indexing

[0057] Vector indexes are a data structure needed for vector retrieval, which can improve the performance of vector approximate nearest neighbor (ANN) retrieval.

[0058] (5) Index building

[0059] Index building involves obtaining feature vectors from various types of unstructured data using inference representation techniques and writing them into the engine to complete the index building process.

[0060] (6) Vector Recall / Vector Retrieval

[0061] Vector recall / vector retrieval refers to using the features of an online request to transform it into a query vector, and then retrieving vector data from a vector index that is similar to the query vector. In this context, raw, unstructured data with similar meanings are close in distance within the vector space.

[0062] (7) Materials

[0063] Creative materials refer to various resources and elements designed to attract potential customers, increase brand awareness, boost website traffic, or promote sales. Materials can take the form of images, text, web pages, videos, or animations. They can be applied to search, recommendation, or advertising scenarios to enhance user engagement and conversion rates. For example, materials can be placed on a website or within web pages.

[0064] Materials can primarily consist of content data and scalar filtering conditions. Content data can include one or more of the following: title, tags, the smallest granular identifier of the content, promotional purpose, promotional application name, creative, copy, images, and videos. Scalar filtering conditions refer to the filtering expressions that must be met during material retrieval; these can also be combinations of multiple filtering expressions. Scalar filtering conditions ensure that the content data accurately reaches the intended target audience.

[0065] (8) Pre-filter

[0066] Pre-filtering refers to scalar filtering performed before vector retrieval.

[0067] (9) Post-filter

[0068] Post-filtering refers to scalar filtering performed after vector retrieval.

[0069] (10) Vector retrieval with scalar filtering

[0070] Vector retrieval with scalar filtering refers to finding the vector x in the material library that is closest to q while meeting the filtering condition f, given a query vector q and a scalar filtering condition f.

[0071] (11) Vector Database

[0072] A vector database is a low-level library used to store vector data.

[0073] (12) Inverted Index

[0074] An inverted index is an index storage format that maps words appearing in a document to a list of documents containing those words.

[0075] The technical solution provided in this application will be described below.

[0076] For example, Figure 1 A schematic diagram illustrating an application scenario provided by an embodiment of this application is shown. For example... Figure 1 As shown, it demonstrates a recommendation advertising service system implemented using a vector retrieval engine. Figure 1 In this process, advertiser-provided materials can be pre-indexed. Then, during ad delivery, user characteristics are converted into user vector representations, and retrieval is performed within the pre-built index using appropriate scalar filtering conditions. The retrieved results are then delivered to the user. Figure 1 The scenario shown can be extended to search promotion (search, recommendation, advertising) businesses. Online search requests from clients can include user characteristics (such as user queries, user profiles, and contextual information viewed by the user). Search / advertising / recommendation scenarios involve multi-dimensional filtering conditions, such as scalar filtering conditions like targeting constraints and keyword matching.

[0077] For example, as shown in Table 1, consider the following advertising scenarios: Advertiser A advertises on "App 1," expecting their target audience to reside in Beijing, Shenzhen, or Shanghai, have "App 2" or "App 3" installed on their phones, be male, be between 20 and 30 years old, and have previously or currently searched for the keyword "travel." Advertiser B advertises on "App 4," expecting their target audience to reside in Beijing or Shenzhen, have "App 1" or "App 3" installed on their phones, with no restrictions on gender or age, and have previously or currently searched for the keyword "travel." The terms "can be bypassed, weak targeting" in Table 1 indicate that these conditions are not necessarily required.

[0078]

[0079] Table 1

[0080] For user A, who has the following personal scalar attributes: male, residing in Beijing, has installed app 2, and has searched for "travel", their ad set is empty. This is because they do not meet advertiser A's requirement of "gender-reverse targeting: male" or advertiser B's requirement of "installed app list-targeting: {app 1, app 3}". For user B: female, 25 years old, residing in Shenzhen, has installed apps 2 and 3, and has searched for "travel", their ad set is {app 1, app 4}.

[0081] As can be seen from the above description, multi-dimensional filtering conditions exist in search / recommendation / advertising scenarios, such as scalar filtering conditions like targeting constraints and keyword matching. Quickly retrieving vector similarity results that meet scalar filtering conditions is a challenge in the industry.

[0082] Building upon this challenge, embodiments of this application provide a vector retrieval architecture with scalar filtering conditions. For example... Figure 2 As shown, this vector retrieval architecture can be divided into offline and online components. In the offline component, data within the materials is processed, creating indexes for scalar filtering conditions and vector indexes for the material content data, which are then stored separately. In the online component, user queries are encoded into vectors, and these vectors, along with the corresponding scalar filtering conditions, are input into the system for vector retrieval and scalar filtering, ultimately returning the top N data points that meet the requirements. Figure 2 As shown in the architecture, the indexes for scalar filtering conditions and the vector indexes for material content data are stored separately. Therefore, an additional external component is needed to store the indexes for scalar filtering conditions in this system. Furthermore, scalar filtering needs to be performed separately during online retrieval. This leads to a significant increase in the overall latency and complexity of online retrieval as the amount of scalar filtering data and the complexity of scalar conditions increase. Simultaneously, offline index building time and memory usage will also gradually increase. Moreover, since there is no interaction between scalar filtering and vector retrieval, the retrieved nearest neighbor results contain a large number of materials that do not meet the scalar filtering conditions. These materials are pre- / post-filtered in the ANN stage, adding unnecessary storage and computation to the vector engine, resulting in no search results or insufficient recall, and wasting bandwidth.

[0083] In view of this, this application proposes another vector retrieval system with scalar filtering conditions. This system can transform material content data and scalar filtering conditions into a vector representation and construct a vector index. Furthermore, it can utilize the vector dimension to control and learn the clustering information of the scalar filtering conditions, performing vector retrieval by partition, thus achieving high-performance vector retrieval with complex scalar soft filtering functions under large data volumes. This eliminates the need for an additional external component to store the index of scalar filtering conditions, achieving a unified architecture and significantly improving the retrieval performance and effectiveness of vector retrieval with scalar filtering conditions.

[0084] For example, Figure 3 This illustration shows a schematic diagram of a vector retrieval technique with scalar filtering conditions provided in an embodiment of this application. Figure 3As shown, this technical concept is also divided into offline and online parts. In the offline part, a vector generation module that integrates scalar filtering conditions and material content data can be used to generate a vector representation, enabling interaction between scalars and vectors. Then, a vector index structure can be built, and the corresponding vector representations can be loaded and stored in the vector index. This allows for automatic scalar partitioning and filtering using clustering information from the vector space distribution. Because the vector representation generated by the scalar filtering integration module incorporates both scalar filtering conditions and material content data (i.e., the vector representation can represent both scalar filtering conditions and material content data), a vector representation in the stored vector index can be associated with a material. A material can include material content data and at least one scalar filtering condition. This eliminates the need to separately store the index for scalar filtering conditions, thus eliminating the need for additional external components, saving disk and memory storage space for the index, and improving the latency of offline index construction. Furthermore, since scalar filtering conditions do not require additional storage of inverted indexes and do not need to consider the complexity and number of scalars, storage resource consumption can be reduced, further expanding the retrieval scale. In other words, by controlling the complexity of scalars and the fineness of filtering in the vector dimension, the retrieval system can support large-scale complex scalar filtering conditions and further expand the retrieval scale.

[0085] Similarly, in the online component, a query generation module that integrates scalar filtering can vectorize user features (such as one or more of the user's query, user profile, and browsing context information) and corresponding scalar filtering conditions to merge them into a single vector representation. Then, an ANN retrieval method can be used to search within a vector index with scalar soft filtering capabilities, ultimately returning the top N data points that meet the requirements. For example, user features can refer to data that characterizes user profiles and / or behaviors, enabling personalized recommendations, search result optimization, and targeted advertising. Since user features and scalar filtering conditions are merged and transformed into a single vector representation during retrieval, a single search can simultaneously filter both content data and scalar filtering conditions. This unified input standard (i.e., both scalar filtering conditions and content data are input in vector representation) resolves the inconsistency between vector and scalar retrieval targets, reduces losses before and after scalar filtering, and improves retrieval accuracy. Meanwhile, by using a unified input standard, the number and complexity of scalar filtering conditions do not need to be considered during the retrieval stage, thus solving the problem of increased latency and complexity in online retrieval and improving retrieval performance.

[0086] The following describes the specific implementation process of the above technical concept.

[0087] For example, Figure 4 This diagram illustrates the architecture of a vector retrieval system with scalar filtering conditions provided in an embodiment of this application. Figure 4 As shown, the vector retrieval system 400 provides both offline and online services. The offline service is a vector engine offline index building service, primarily used to build vector indexes with scalar soft filtering capabilities, constructing vector indexes based on implicit vector partitioning (i.e., implicit scalar partitioning through vectors) soft filtering. The online service is a vector online retrieval service component for the search promotion (search, recommendation, advertising) platform, primarily used to process online search / recommendation / advertising requests. It can generate vector representations incorporating scalar filtering conditions, load vector indexes with scalar soft filtering capabilities, and complete vector retrieval with scalar filtering functionality.

[0088] The vector retrieval system 400 can be configured with a material library 410, a vector generation module 420 with scalar filtering, an index building module 430, and a vector index 440 with scalar soft filtering functionality, so that the vector retrieval system 400 can provide offline services. The material library 410 is mainly used to store materials. Each material may include: material content data and at least one scalar filtering condition. For example, please refer to... Figure 5 , Figure 5 (A) shows a material provided by advertiser A, which contains Figure 5 The scalar filtering conditions and material content data described in (B). Among them, Figure 5 (B) represents the data after normalizing the scalar filtering conditions and material content data in the material. This facilitates subsequent unified vectorization of both, converting them into a single vector representation. In some embodiments, when normalizing the scalar filtering conditions and material content data, the scalar filtering conditions can be converted into a fixed format (e.g., Boolean expressions or JSON format). For example, the scalar filtering conditions can be converted into Boolean expressions, transforming the filtering logic into operators such as In, AND, OR, and NOT. When normalizing the material content data, the material content data can be identified (ID), ultimately outputting a series of IDs. These IDs can, but are not limited to, form an ID list. For example, when an element in the material content data is already an ID (e.g., ...), ... Figure 5 When an element has an ID such as the creative ID or task ID in (A), it can be mapped to an ID map to obtain the corresponding ID; however, when an element is not an ID (e.g., Figure 5When considering the promotional purpose (e.g., in (A)), you can first perform word segmentation and then map the segmented data to an ID map to obtain the corresponding IDs. This allows you to map different elements to a unified space. It should be understood that you can also standardize the material content data through methods such as enumeration, depending on the actual situation. No specific method is specified here.

[0089] The vector generation module 420 for scalar filtering integration is mainly used to vectorize scalar filtering conditions and material content data (i.e., scalar filtering conditions and material content data within a material) that have a relationship. This allows the scalar filtering conditions to be vectorized and, together with the material content data, generated into a vector representation, thus achieving the fusion of scalar filtering conditions and material content data. Specifically, the vector generation module 420 generates a vector representation for each material it processes. Each vector representation generated by the scalar filtering module 420 represents the material content data of a material, as well as the scalar filtering conditions for that material. In the scalar filtering module 420, scalar filtering conditions and material content data with a relationship can interact in various ways, including intra-scalar interaction, intra-material interaction, and scalar-vector interaction. This allows them to learn from each other, facilitating the grouping of similar ideas and scalars when building an index. In some embodiments, see further... Figure 5 ,like Figure 5 As shown in (C), the vector generation module 420 for scalar filtering can include: a binarization layer 421, an identifier layer 422, a logical layer 423, a data encoder 424, and a fusion block 425. The binarization layer 421 is mainly used to binarize the range-type conditions in the scalar filtering conditions to ID the range-type conditions. The identifier layer 422 is mainly used to process the match-type conditions in the scalar filtering conditions. For example, match-type conditions can be segmented into words, and the segmented data can be mapped to an identifier map to ID the match-type conditions. After both range-type and match-type conditions are ID-ified, they can be merged, for example, by concatenation, to obtain the ID list of the scalar filtering conditions. The logic layer 423 is mainly used to process the ID list of scalar filtering conditions to perform scalar internal interaction, enabling the data in the scalar filtering conditions to learn information from each other, thereby obtaining the vector representation Z of the scalar filtering conditions. rFor example, logic layer 423 may include a conjunction operator network and a disjunction operator network. The conjunction operator network can be used to perform interactions between different scalars, and the disjunction operator network can be used to perform interactions between the same scalar. Figure 5 h shown in (C) j This is a hidden layer; when j=1, it is the input layer; W tj This represents the network parameter matrix. The data encoder 424 is mainly used to process the LD list of material content data to enable internal interaction among the material content data, allowing the data within the material content data to learn from each other and thus obtain the vector representation Z of the material content data. d For example, in the merged vector representation Z r and Z d In such cases, a weighted approach can be used to merge the two components to improve their fusion effect and avoid the reduced effectiveness that can result from a direct merger. Figure 5 In (C), α represents the vector representation weight of the scalar filtering condition, and 1-α represents the vector representation weight of the material content data. α can be learned or is a hyperparameter; this is not limited here. The fusion module 425 is mainly used for the merged vector representation Z. r and Z d The process involves interaction between scalar filtering conditions and material content data, allowing them to learn from each other and ultimately obtain a vector representation that integrates both scalar filtering conditions and material content data. In this embodiment, both the logic layer 423 and the fusion module 425 can be, but are not limited to, multilayer perceptrons. Furthermore, using a logic layer 423 that supports both conjunction and disjunction paradigms can enhance the interpretability of the results. In some embodiments, during the training process of the vector generation module 420, the objective function (i.e., the loss function) L can be used to control the loss ratio between the scalar filtering conditions and the material content data for deep representation learning, thereby simultaneously satisfying business objectives and relevant rules and improving delivery effectiveness. In this embodiment, after processing the materials in the material library through the vector generation module 420, vector representations of the corresponding materials can be obtained, each vector representation integrating the corresponding material content data and scalar filtering conditions. For example, Figure 5 The (D) represents a series of vector representations.

[0090] The index building module 430 primarily uses industry-standard vector algorithms (such as inverted file (IVF) classes, graphs, etc.) to process the vector representation generated by the vector generation module 420 to construct a vector index. Regardless of the vector algorithm used, the index building module 430's principle for constructing the vector index is to aggregate similar vectors together, facilitating quick retrieval of regions similar to the target vector and traversing within those regions. For example, the index building module 430 can process the vector representation output by the vector generation module 420 using clustering algorithms such as K-Nearest Neighbors (KNN) to learn the structure of the corresponding vector index. For example, the structure of the vector index can include nodes and edges. Nodes can indicate cluster centers, and edges can indicate the relationships between nodes. After obtaining the structure of the vector index, the vector representation output by the vector generation module 420 can be loaded into the corresponding nodes (also called "buckets") of the vector index. Within each bucket, the distance between the bucket's center and its internal vector representation is less than the distance between the internal vector representation and the centers of other buckets. For example, assuming there are 100,000 items in the material library, 100,000 vector representations can be obtained. Then, the structure of the vector index can be learned from all or part of these 100,000 vector representations to obtain the structure of the vector index. For example, the index structure can be a tree structure, where the first-level index includes the index values ​​of multiple cluster centers in multiple clusters, and the second-level index includes the index values ​​of each cluster. After obtaining the structure of the vector index, the 100,000 vector representations can be loaded into the corresponding buckets in the vector index, meaning each vector representation is assigned to a corresponding bucket. For instance, the index building module 430 can first parse the vector representations output by the vector generation module 420; then, it can train an index structure model of type IVF or similar using the parsed data; finally, it can construct the vector index using the trained index structure model. It should be understood that because the vector representation output by the vector generation module 420 incorporates interactions between the scalar filtering conditions, the material content data, and the data itself, the vector representation output by the vector generation module 420 integrates the features of both the scalar filtering conditions and the material content data. This allows for the grouping of similar ideas and scalars together during index construction. For example, ... Figure 5 As shown in (E), only ads with similar scalar filtering conditions and behaviors will be grouped into the same bucket.

[0091] The vector index 440 with scalar soft filtering function is mainly used to store the index structure built by the index building module 430. It can be understood as an index data storage module.

[0092] Additionally, the vector retrieval system 400 can be configured with an interface 450, a query generation module 460 incorporating scalar filtering, and a vector retrieval module 470, so that the vector retrieval system 400 can provide online services. The interface 450 is primarily used to receive and process online retrieval requests from clients. These online retrieval requests may include user characteristics. Of course, the online retrieval requests may also include scalar filtering conditions and / or filtering strengths. For example, please refer to... Figure 6 , Figure 6 (A) illustrates an online search request that includes user characteristics, scalar filtering conditions, and filtering strength.

[0093] The query generation module 460, which integrates scalar filtering, is primarily used to transform online retrieval requests into vector representations. This module may include a feature processing module 361, a scalar processing module 362, and a representation model (i.e., an embedding model) 363. The feature processing module 361 preprocesses the data in the online retrieval request, such as deduplication and tokenization. The scalar processing module 362 primarily converts the scalar filtering conditions in the online retrieval request into formats, such as performing Boolean expression conversion and transforming the filtering logic into operator descriptions like IN, AND, OR, and NOT. Through the feature processing module 361 and the scalar processing module 362, the online retrieval request can be transformed into a standardized data representation. The embedding model 363 primarily transforms the data processed by the feature processing module 361 and the scalar processing module 362 into a vector representation. When the online search request includes a filtering intensity, the query generation module 460 can multiply its generated vector representation, which integrates user features and scalar filtering conditions, with the filtering intensity to obtain the desired user vector representation. For example, the working principle of the query generation module 460 is similar to that of the aforementioned vector generation module 420, as detailed above, and will not be repeated here. The vector retrieval module 470 is mainly used to complete the ANN retrieval and return TOP-K results. For example, see [link to relevant documentation]. Figure 6 ,like Figure 6 As shown in (B), when there are multiple online search requests, the query generation module 460 can process each online search request separately to obtain the user vector representation corresponding to each online search request. Then, as... Figure 6 As shown in (C), ANN retrieval is performed on each user vector representation. Finally, as... Figure 6As shown in (D), for each online search request, Top-K recall results can be retrieved and returned to the client. As described in the online service phase, scalar filtering conditions and user characteristics are input into the query generation module to obtain a vector representation of the user with integrated scalar filtering conditions. Furthermore, during the vector retrieval scanning process, data that basically meets the scalar filtering conditions is found through vector retrieval, achieving controllability of scalar filtering. Thus, by uniformly converting user characteristics and scalar filtering conditions in the online search request into vector representations, it is unnecessary to calculate the vector representation of material content data and parse the scalar filtering conditions separately in two modules, accelerating the initial search preparation work and improving performance. In addition, when the online search request includes filtering intensity, the strength of scalar filtering can be controlled using the filtering intensity to achieve soft scalar filtering.

[0094] The following sections will introduce the specific implementation processes of offline and online services in the Vector Retrieval System 400.

[0095] (1) Specific implementation process of offline service

[0096] For example, Figure 7 A flowchart illustrating a data processing method provided in an embodiment of this application is shown. Figure 7 As shown, it can be understood that this method can be executed by any device, equipment, platform, or cluster of devices with computing and processing capabilities. For example, the method can be executed by a data processing device, which can be implemented by software and / or hardware, and can be, but is not limited to, configured in a cluster of computing devices containing at least one computing device; typically, it can be configured in a server. For ease of description, the following description will focus on a server as the executing entity. Figure 7 As shown, the data processing method may include the following steps:

[0097] S701. Obtain a first material from the material library. The first material includes: first material content data and a first scalar filter condition, wherein the first scalar filter condition is a filter expression condition that must be satisfied during the retrieval of the first material content data.

[0098] In this embodiment, the server can load materials from a material library; of course, the server can also obtain the first material from other places, which is not limited here. A material may include the material content data of the material and the scalar filtering conditions of the material. The material content data and the scalar filtering conditions in a material are related. For example, a material may have at least one scalar filtering condition, that is, at least one first scalar filtering condition. Furthermore, the scalar filtering conditions in a material are the filtering expression conditions that must be satisfied during the retrieval of the material content data in that material. For example, such as... Figure 5As shown in (A), it represents a material provided by advertiser A; the specific scalar filtering conditions and material content data in this material can be seen as follows: Figure 5 As shown in (B).

[0099] S702. The first material content data and the first scalar filtering condition are vectorized using a neural network model to obtain a first vector representation, wherein the first vector representation is used to characterize the first material content data and the first scalar filtering condition.

[0100] In this embodiment, after obtaining the first material, the server can use a neural network model to uniformly vectorize the first material content data and the first scalar filtering conditions to obtain a first vector representation. That is, the first material content data and the first scalar filtering conditions can be input into the neural network model for processing to obtain the first vector representation. The first vector representation is used to characterize the first material content data and the first scalar filtering conditions. In this embodiment, the neural network model can fuse the first material content data and the first scalar filtering conditions together to form a unified vector representation (i.e., the first vector representation); however, the two are not simply concatenated together, but need to be mutually integrated, such as through interactive learning.

[0101] As one possible implementation, such as Figure 8 As shown, the process of obtaining the first vector may include: in S801, vectorizing the first material content data to obtain a third vector representation. Specifically, the first material content data may first be identified to convert it into normalized data, thereby obtaining at least one first identifier (this process can be referred to above). Figure 4 (The relevant descriptions in the text are omitted here). Then, interactive learning is performed on at least one first identifier to obtain a third vector representation. In this way, interactive learning can be achieved within the material content data, so that the data in the material content data can learn information from each other, improve the accuracy of the obtained vector representation, and facilitate the subsequent aggregation of similar data in the material content data. For example, the neural network model used in S702 can be, but is not limited to, the aforementioned Figure 4 In the vector generation module 420 described above, the third vector representation can be obtained by processing the first identifier through the data encoder 424. In S802, the first scalar filtering condition is vectorized to obtain the fourth vector representation. Specifically, the first scalar filtering condition can be format-converted, for example, converted into a Boolean logical expression, to obtain the first expression. Then, the first expression is further tokenized to obtain at least one second identifier (this process can be referred to above). Figure 4(The relevant descriptions in [the original text] will not be repeated here). Finally, interactive learning is performed on at least one second identifier to obtain a fourth vector representation. In this way, the scalars in the scalar filtering conditions can learn each other's information, improving the accuracy of the obtained vector representation, and facilitating the subsequent aggregation of similar scalar data, reducing the loss after merging the scalar filtering conditions. For example, the neural network model used in S702 can be, but is not limited to, the aforementioned [model / model]. Figure 4 In the vector generation module 420 described above, the fourth vector representation can be obtained by processing at least one second identifier through the logic layer 423. In S803, the third and fourth vector representations are interactively learned to obtain the first vector representation. This allows material content data and scalar filtering conditions to learn each other's information, facilitating the grouping of similar types of ideas and scalars when building an index. For example, the neural network model used in S702 can be, but is not limited to, the aforementioned... Figure 4 In the vector generation module 420 described herein, the first vector representation can be obtained by processing the third and fourth vector representations through the fusion module 425. In some embodiments, during the processing of the third and fourth vector representations using a neural network model, the weights of the third and fourth vector representations are different. Therefore, they are merged using a weighted method to improve the fusion effect and avoid the performance degradation caused by direct merging. In some embodiments, the third vector representation can be understood as... Figure 5 The vector representation Z in (C) r The fourth vector representation can be understood as... Figure 5 The vector representation Z in (C) r .

[0102] S703. Load the first vector representation into the vector index.

[0103] In this embodiment, after obtaining the first vector representation, the server can load the first vector representation into the vector index for storage. In some embodiments, the structure of the vector index may include nodes and edges. Nodes can be used to indicate cluster centers, and edges can be used to indicate the affiliation relationships between nodes. When loading the first vector representation into the vector index, the first vector representation can be matched with the nodes in the vector index structure according to the node hierarchy until the most similar node is matched, and the first vector representation is stored in that node. In some embodiments, the server can process each material in the material library to obtain a vector representation for each material. Then, the server can use industry-standard vector algorithms (such as IVF classes, graphs, etc.) to process all or a portion of the obtained vector representations to learn the structure of the vector index. After learning the structure of the vector index, the server can load the vector representation corresponding to each material into the vector index. The process of learning the structure of the vector index is detailed in the aforementioned section. Figure 4 The description of the index building module 430 will not be repeated here.

[0104] In this way, by uniformly converting material content data and scalar filtering conditions into vector representations and constructing a vector index, it is no longer necessary to store the scalar filtering condition index separately, nor is it necessary to consider the number and complexity of scalar filtering conditions. This eliminates the need for additional external components, saving disk and memory storage space for the index and improving the latency of offline index construction. Furthermore, with this index structure, during the online retrieval phase, by using a unified input standard (i.e., both scalar filtering conditions and material content data are input as vector representations), and through a single retrieval, it is possible to simultaneously query and filter both material content data and scalar filtering conditions. This solves the problem of inconsistency between vector and scalar retrieval targets, reduces losses before and after scalar filtering, and improves retrieval accuracy. Simultaneously, because of the unified input standard, the number and complexity of scalar filtering conditions do not need to be considered during the retrieval phase, thus solving the problem of increased latency and complexity in online retrieval and improving retrieval performance.

[0105] (2) Specific implementation process of online services

[0106] For example, Figure 9 A flowchart illustrating a data processing method provided in an embodiment of this application is shown. Figure 9As shown, it can be understood that this method can be executed by any device, equipment, platform, or cluster of devices with computing and processing capabilities. For example, the method can be executed by a data processing device, which can be implemented by software and / or hardware, and can be, but is not limited to, configured in a cluster of computing devices containing at least one computing device; typically, it can be configured in a server. For ease of description, the following description will focus on a server as the executing entity. Figure 9 As shown, the data processing method may include the following steps:

[0107] S901. Receive a retrieval request from a client, the retrieval request including user characteristics.

[0108] In this embodiment, when a user uses the client, the client can send a search request to the server. This search request may include user characteristics, such as the user's query, user profile, and contextual information the user has viewed. Of course, the search request may also include scalar filtering conditions and / or filtering strength. This is not limited here. For example, user characteristics can refer to data that can characterize a user profile and / or behavior. Personalized recommendations, search result optimization, and targeted advertising can be achieved through user characteristics.

[0109] S902. Perform vectorization processing on user features to obtain user vector representations.

[0110] In this embodiment, the server can vectorize the user features in the retrieval request to obtain a user vector representation. For example, this can be achieved through the aforementioned... Figure 4 The query generation module 460 in the system converts user features into vector representations, or converts user features into vector representations through other neural network models, etc. Additionally, when the search request also includes scalar filtering conditions (hereinafter referred to as "second scalar filtering conditions"), the server can use neural network models (e.g., the aforementioned...) Figure 4 The query generation module 460 described herein vectorizes user features and the second scalar filtering conditions to obtain a user vector representation. This user vector representation can then be used to characterize the user features and the second scalar filtering conditions. Furthermore, when the retrieval request includes a filtering strength, the server, after obtaining the user vector representation, can update it using the filtering strength, for example, by multiplying the two, to control the strength of the scalar filtering and achieve scalar soft filtering.

[0111] S903. Based on the user vector representation, perform a search in the vector index to obtain at least one vector representation corresponding to the user vector representation.

[0112] In this embodiment, the server can process the user vector representation and vector index based on algorithms such as similarity or clustering to retrieve at least one vector representation corresponding to the user vector representation from the vector index (which is constructed offline). A vector representation in the vector index can be used to characterize the material content data and scalar filtering conditions contained in a material. For example, the vector representation in the vector index can also be called a "material vector representation." Furthermore, the similarity between the retrieved vector representation and the user vector representation meets preset requirements. For example, such as... Figure 10 As shown, during the retrieval process, in S1001, an IVF-type index is first used to retrieve the list of buckets to be scanned. In S1002, for each bucket to be scanned, each vector representation within the bucket is traversed. In S1003, algorithms such as cosine similarity are used to calculate the vector similarity between the user's vector representation and the corresponding vector representation in the bucket. In S1004, it is determined whether the recall results need to meet a similarity threshold, for example, the similarity must be greater than 0.8. If not, proceed to S1005; otherwise, proceed to S1006. In S1005, the material content data associated with the K most similar vector representations is returned, i.e., the material content data represented by the K vector representations is returned. In S1006, vector representations with similarity less than the similarity threshold are filtered out. In S1007, the material content data associated with the vector representations that meet the similarity threshold condition is returned, i.e., the material content data represented by the corresponding vector representations is returned. In this way, the required results are retrieved from the index structure.

[0113] S904. Return to the client the material content data represented in at least one vector representation, or return to the client the identifier of at least one vector representation.

[0114] In this embodiment, after retrieving the required vector representation, the server can query the corresponding material from the material library using the identifier of the vector representation and return the material content data of the queried material to the client. Each vector representation can have a unique identifier, and each vector representation identifier can have a mapping relationship with a material. Alternatively, the server can directly return the identifier of the queried vector representation to the client, allowing the client to query the corresponding material content data itself; the specific method depends on the actual situation and is not limited here.

[0115] In this way, since a single search can simultaneously query and filter both material content data and scalar filtering conditions, the inconsistency between vector and scalar search targets is resolved, losses before and after scalar filtering are reduced, and search accuracy is improved. Furthermore, the number and complexity of scalar filtering conditions do not need to be considered during the search phase, thus addressing the issues of increased latency and complexity in online searches and improving search performance.

[0116] The effects of the solutions provided in the embodiments of this application will be described below.

[0117] like Figure 11 As shown, a comparison between the search method provided in this solution and the search methods in related technologies reveals that, for example... Figure 11 As shown in (A), when the number of indexes is 5 million, the queries per second (QPS) is 500, and the recall is 200, the online retrieval latency in related technical solutions increases significantly with the increase of the number of scalar filter conditions. However, in this solution, the online retrieval latency is independent of the number of scalar filter conditions. Figure 11 As shown in (B), when the number of indexes is 5 million, the number of scalar filtering conditions is 20, and the number of recalled results is 200, the online retrieval latency in the relevant technical solutions increases significantly with the increase of the complexity of the scalar filtering logic. However, in this solution, the online retrieval latency is independent of the complexity of the scalar filtering logic.

[0118] like Figure 11 As shown in (C), in the scenarios of search advertising and display advertising, the memory usage is high when using an external component to store scalar filtering conditions (i.e., storing them through an inverted index). However, this solution eliminates the storage of the inverted index (i.e., it does not require an additional external component to store the index of scalar filtering conditions), and the memory space usage is significantly reduced compared with the retrieval methods in related technologies.

[0119] like Figure 11 As shown in (D), among the 100 recalled data points, the recall results obtained through this solution that satisfy scalar filtering conditions A and B are higher than those obtained through related technologies. In other words, the scalar condition hit rate of the offline recall results is also significantly improved, achieving the function of filtering using vector retrieval, significantly reducing the loss before and after scalar filtering, and improving retrieval performance.

[0120] It is understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In addition, the various embodiments or technical features involved in the embodiments described above can be combined according to the actual situation, and the combined solution is still within the protection scope of this application.

[0121] Based on the methods in the above embodiments, this application also provides a data processing apparatus.

[0122] For example, Figure 12 A schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application is shown.Figure 12 As shown, the data processing device 1200 includes an acquisition module 1201 and a processing module 1202. The acquisition module 1201 is used to acquire a first material, which includes first material content data and a first scalar filtering condition. The processing module 1202 is used to vectorize the first material content data and the first scalar filtering condition using a neural network model to obtain a first vector representation, wherein the first vector representation is used to characterize the first material content data and the first scalar filtering condition. The processing module 1202 is also used to load the first vector representation into a vector index.

[0123] In some embodiments, when the processing module 1202 performs vectorization processing on the first material content data and the first scalar filtering conditions through a neural network model to obtain a first vector representation, it is specifically used to: perform vectorization processing on the first material content data to obtain a third vector representation; perform vectorization processing on the first scalar filtering conditions to obtain a fourth vector representation; and perform interactive learning on the third vector representation and the fourth vector representation to obtain the first vector representation.

[0124] In some embodiments, when the processing module 1202 performs vectorization processing on the first material content data to obtain a third vector representation, it is specifically used to: perform identifier processing on the first material content data to obtain at least one first identifier; and perform interactive learning on the at least one first identifier to obtain a third vector representation.

[0125] In some embodiments, when the processing module 1202 performs vectorization processing on the first scalar filtering condition to obtain a fourth vector representation, it is specifically used to: perform format conversion on the first scalar filtering condition to obtain a first expression; perform identifier processing on the first expression to obtain at least one second identifier; and process the at least one second identifier to obtain a fourth vector representation.

[0126] In some embodiments, the weight ratios of the third vector representation and the fourth vector representation are different.

[0127] In some embodiments, after loading the first vector representation into the vector index, the processing module 1202 is further configured to: receive a retrieval request from a client, the retrieval request including user features; perform vectorization processing on the user features to obtain a user vector representation; perform a retrieval in the vector index based on the user vector representation to obtain at least one vector representation corresponding to the user vector representation; return the material content data represented by at least one vector representation to the client, or return the identifier of at least one vector representation to the client.

[0128] In some embodiments, the retrieval request further includes a second scalar filtering condition. In this case, when the processing module 1202 performs vectorization processing on the user features to obtain a user vector representation, it specifically performs: vectorization processing on the user features and the second scalar filtering condition using a neural network model to obtain a user vector representation, wherein the user vector representation is used to characterize the user features and the second scalar filtering condition.

[0129] In some embodiments, the retrieval request further includes: filtering strength. At this time, before performing the retrieval in the vector index based on the user vector representation, the processing module 1202 is also used to: update the user vector representation based on the filtering strength.

[0130] In some embodiments, the structure of the vector index includes nodes and edges, where nodes represent cluster centers and edges represent the affiliation relationships between nodes.

[0131] In some embodiments, Figure 12 Both the acquisition module 1201 and the processing module 1202 shown can be implemented in software or in hardware. For example, the implementation of the acquisition module 1201 will be described below. Similarly, the implementation of the processing module 1202 can refer to the implementation of the acquisition module 1201.

[0132] As an example of a software functional unit, module 1201 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, module 1201 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0133] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0134] As an example of a hardware functional unit, the acquisition module 1201 may include at least one computing device, such as a server. Alternatively, the acquisition module 1201 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0135] The multiple computing devices included in the acquisition module 1201 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the acquisition module 1201 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the acquisition module 1201 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0136] It should be noted that, in other embodiments, the acquisition module 1201 can be used to execute any step in the data processing method described in the above embodiments, and the processing module 1202 can be used to execute any step in the data processing method described in the above embodiments. Furthermore, the steps implemented by the acquisition module 1201 and the processing module 1202 can be specified as needed, and different steps in the data processing method described in the above embodiments can be implemented by the acquisition module 1201 and the processing module 1202 respectively. Figure 12 The data processing device 1200 shown has all the functions of the data processing device.

[0137] This application also provides a computing device 1300. For example... Figure 13 As shown, the computing device 1300 includes a bus 1302, a processor 1304, a memory 1306, and a communication interface 1308. The processor 1304, the memory 1306, and the communication interface 1308 communicate with each other via the bus 1302. The computing device 1300 can be a server or an electronic device. It should be understood that this application does not limit the number of processors and memories in the computing device 1300.

[0138] Bus 1302 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 13 The bus 1304 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1304 may include a path for transmitting information between various components of the computing device 1300 (e.g., memory 1306, processor 1304, communication interface 1308).

[0139] The processor 1304 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0140] The memory 1306 may include volatile memory, such as random access memory (RAM). The processor 1304 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0141] The memory 1306 stores executable program code, and the processor 1304 executes the executable program code to implement the aforementioned functions. Figure 12 The functions of the acquisition module 1201 and processing module 1202 shown are implemented to realize the data processing method described in the above embodiments. That is, the memory 1306 stores instructions for executing the data processing method described in the above embodiments.

[0142] Alternatively, the memory 1306 stores executable code, and the processor 1304 executes the executable code to implement the aforementioned functions respectively. Figure 12 The data processing apparatus 1200 shown herein performs the functions of the data processing method described in the above embodiments. That is, the memory 1306 stores instructions for executing the data processing method described in the above embodiments.

[0143] The communication interface 1303 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1300 and other devices or communication networks.

[0144] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be an electronic device such as a desktop computer, a laptop computer, or a smartphone.

[0145] like Figure 14 As shown, the computing device cluster includes at least one computing device 1300. The memory 1306 of one or more computing devices 1300 in the computing device cluster may store the same instructions for performing the data processing methods described in the above embodiments.

[0146] In some possible implementations, the memory 1306 of one or more computing devices 1300 in the computing device cluster may also store partial instructions for executing the data processing method described in the above embodiments. In other words, a combination of one or more computing devices 1300 can jointly execute instructions for executing the data processing method described in the above embodiments.

[0147] It should be noted that the memory 1306 in different computing devices 1300 within the computing device cluster can store different instructions, each used to execute the aforementioned instructions. Figure 12 This illustrates some of the functions of the data processing device 1200. Specifically, the instructions stored in the memory 1306 of different computing devices 1300 can implement the functions of one or more modules in the acquisition module 1201 and processing module 1202.

[0148] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 15 One possible implementation is shown. For example... Figure 15 As shown, the two computing devices 1300A and 1300B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 1306 in computing device 1300A stores instructions for executing the functions of the acquisition module 1201. Simultaneously, the memory 1306 in computing device 1300B stores instructions for executing the functions of the processing module 1202.

[0149] It should be understood that Figure 15The functions of computing device 1300A shown can also be performed by multiple computing devices 1300. Similarly, the functions of computing device 1300B can also be performed by multiple computing devices 1300.

[0150] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 14 and Figure 15 The connection method of the computing device cluster is different in that the memory 1306 of one or more computing devices 1300 in the computing device cluster can store the same instructions for executing the methods in the above embodiments.

[0151] In some possible implementations, the memory 1306 of one or more computing devices 1300 in the computing device cluster may also store partial instructions for executing the aforementioned data processing method. In other words, a combination of one or more computing devices 1300 can jointly execute instructions for executing the aforementioned data processing method.

[0152] Based on the methods in the above embodiments, this application provides a computer-readable storage medium including computer program instructions. When the computer program instructions are executed by a computing device, the computing device performs the methods in the above embodiments; or, when the computer program instructions are executed by a cluster of computing devices, the cluster of computing devices performs the methods in the above embodiments. Exemplarily, the computer-readable storage medium can be any available medium that the computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0153] Based on the methods in the above embodiments, this application provides a computer program product containing instructions that, when executed by a computing device, cause the computing device to perform the methods in the above embodiments, or, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the methods in the above embodiments.

[0154] Based on the methods in the above embodiments, this application also provides a data processing apparatus, including: an acquisition module and a processing module. The acquisition module is used to receive a retrieval request from a client, the retrieval request including user features and scalar filtering conditions. The processing module is used to vectorize the user features and scalar filtering conditions using a neural network model to obtain a user vector representation, wherein the user vector representation is used to characterize the user features and scalar filtering conditions. The processing module is further used to perform a retrieval among multiple material vector representations based on the user vector representation to obtain at least one material vector representation corresponding to the user vector representation, wherein each material vector representation integrates material content data and scalar filtering conditions for one material, and each material vector representation is used to characterize the material content data of one material. The processing module is further used to return the material content data represented by at least one material vector representation to the client, or to return an identifier of at least one vector representation to the client.

[0155] It should be understood that the above-described device is used to execute the methods in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above methods. The working process of the device can be referred to the corresponding process in the above methods, and will not be repeated here.

[0156] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0157] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0158] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, cloud computing platform, or data center to another website, computer, cloud computing platform, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a cloud computing platform or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0159] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized by, The method comprises: obtaining first material content data and a first scalar filtering condition related to the first material content data; performing vectorization processing on the first material content data and the first scalar filtering condition by a neural network model to obtain a first vector representation, wherein the first vector representation is used to represent the first material content data and the first scalar filtering condition; loading the first vector representation into a vector index.

2. The method of claim 1, wherein, The vectorization processing on the first material content data and the first scalar filtering condition by the neural network model to obtain the first vector representation comprises: performing vectorization processing on the first material content data to obtain a third vector representation; performing vectorization processing on the first scalar filtering condition to obtain a fourth vector representation; performing interactive learning on the third vector representation and the fourth vector representation to obtain the first vector representation.

3. The method of claim 2, wherein, The vectorization processing on the first material content data to obtain the third vector representation comprises: performing identification processing on the first material content data to obtain at least one first identifier; performing interactive learning on the at least one first identifier to obtain the third vector representation.

4. The method according to claim 2 or 3, characterized in that, The vectorization processing on the first scalar filtering condition to obtain the fourth vector representation comprises: performing format conversion on the first scalar filtering condition to obtain a first expression; performing identification processing on the first expression to obtain at least one second identifier; performing processing on the at least one second identifier to obtain the fourth vector representation.

5. The method according to any of claims 2-4, characterized by, The weights of the third vector representation and the fourth vector representation are different.

6. The method according to any one of claims 1 to 5, characterized in that, After loading the first vector representation into the vector index, the method further comprises: receiving a retrieval request from a client, wherein the retrieval request comprises user features; performing vectorization processing on the user features to obtain a user vector representation; based on the user vector representation, performing retrieval in the vector index to obtain at least one vector representation corresponding to the user vector representation; returning, to the client, material content data represented by the at least one vector representation, or returning, to the client, an identifier of the at least one vector representation.

7. The method of claim 6, wherein, The retrieval request further comprises a second scalar filtering condition; The vectorization processing on the user features to obtain a user vector representation comprises: performing vectorization processing on the user features and the second scalar filtering condition by the neural network model to obtain a user vector representation, wherein the user vector representation is used to represent the user features and the second scalar filtering condition.

8. The method according to claim 6 or 7, characterized in that, The retrieval request further comprises a filtering intensity; Before performing retrieval in the vector index based on the user vector representation, the method further comprises: based on the filtering intensity, updating the fourth vector representation.

9. The method according to any of claims 1 to 8, characterized in that, The structure of the vector index comprises nodes and edges, wherein the nodes are used to represent cluster centers, and the edges are used to represent the ownership relationship between the nodes.

10. A data processing method, characterized by, The method comprises: receiving a retrieval request from a client, wherein the retrieval request comprises user features and a scalar filtering condition; vectorize the user feature and the scalar filtering condition by using a neural network model to obtain a user vector representation, wherein the user vector representation fuses the user feature and the scalar filtering condition; based on the user vector representation, search in a plurality of material vector representations to obtain at least one material vector representation corresponding to the user vector representation, wherein each material vector representation is used to represent material content data and a scalar filtering condition of a material; return, to the client, material content data represented by the at least one material vector representation, or return, to the client, an identifier of the at least one vector representation.

11. A data processing apparatus, characterized by comprise: an acquisition module, configured to acquire first material content data and a first scalar filtering condition related to the first material content data; a processing module, configured to vectorize the first material content data and the first scalar filtering condition by using a neural network model to obtain a first vector representation, wherein the first vector representation is used to represent the first material content data and the first scalar filtering condition; the processing module is further configured to load the first vector representation into a vector index.

12. The apparatus of claim 11, wherein, when the processing module vectorizes the first material content data and the first scalar filtering condition by using a neural network model to obtain a first vector representation, the processing module is specifically configured to: vectorize the first material content data to obtain a third vector representation; vectorize the first scalar filtering condition to obtain a fourth vector representation; interactively learn the third vector representation and the fourth vector representation to obtain the first vector representation.

13. The apparatus of claim 12, wherein, when the processing module vectorizes the first material content data to obtain a third vector representation, the processing module is specifically configured to: identify the first material content data to obtain at least one first identifier; interactively learn the at least one first identifier to obtain the third vector representation.

14. The apparatus of claim 12 or 13, wherein, when the processing module vectorizes the first scalar filtering condition to obtain a fourth vector representation, the processing module is specifically configured to: convert the first scalar filtering condition into a first expression; identify the first expression to obtain at least one second identifier; process the at least one second identifier to obtain the fourth vector representation.

15. The apparatus of any of claims 12-14, wherein, weights of the third vector representation and the fourth vector representation are different.

16. The apparatus of any of claims 11-15, wherein, after the processing module loads the first vector representation into the vector index, the processing module is further configured to: receive a search request from a client, wherein the search request comprises a user feature; vectorize the user feature to obtain a user vector representation; based on the user vector representation, search in the vector index to obtain at least one vector representation corresponding to the user vector representation; return, to the client, material content data in a material represented by the at least one vector representation, or return, to the client, an identifier of the at least one vector representation.

17. The apparatus of claim 16, wherein, the search request further comprises a second scalar filtering condition; The processing module is specifically configured to perform vectorization processing on the user features to obtain a user vector representation when the user features are processed. The processing module is configured to perform vectorization processing on the user features and the second scalar filtering condition by using the neural network model to obtain a user vector representation, where the user vector representation is used to represent the user features and the second scalar filtering condition.

18. The apparatus of claim 16 or 17, wherein, The retrieval request further includes a filtering intensity. The processing module is further configured to update the user vector representation based on the filtering intensity before performing retrieval in the vector index based on the user vector representation. The structure of the vector index includes nodes and edges, where the nodes are used to represent cluster centers, and the edges are used to represent the ownership relationship between the nodes.

19. The apparatus of any of claims 11-18, wherein, The method comprises the following steps:

20. A data processing apparatus, characterized in that, The acquisition module is configured to receive a retrieval request from a client, where the retrieval request includes user features and a scalar filtering condition. The processing module is configured to perform vectorization processing on the user features and the scalar filtering condition by using a neural network model to obtain a user vector representation, where the user vector representation is used to represent the user features and the scalar filtering condition. The processing module is further configured to perform retrieval in a plurality of material vector representations based on the user vector representation to obtain at least one material vector representation corresponding to the user vector representation, where each material vector representation is fused with material content data and a scalar filtering condition of a material, and each material vector representation is used to represent the material content data of the material. The processing module is further configured to return the material content data represented by the at least one material vector representation to the client, or return the identifier of the at least one vector representation to the client. The at least one computing device comprises a processor and a memory.

21. A cluster of computing devices, characterized in that, The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to enable the computing device cluster to perform the method according to any one of claims 1-9 or the method according to claim 10. The computer program instructions enable the computing device cluster to perform the method according to any one of claims 1-9 or the method according to claim 10 when the instructions are executed by the computing device cluster, where the computing device cluster comprises at least one computing device.

22. A computer-readable storage medium, characterized in that, The computer program instructions enable the computing device cluster to perform the method according to any one of claims 1-9 or the method according to claim 10 when the instructions are executed by the computing device cluster, where the computing device cluster comprises at least one computing device.

23. A computer program product comprising instructions, characterized in that, ​

Images