Index construction method and device, equipment, medium and computer program product

Abstract

The invention relates to an index construction method, which comprises the following steps of: acquiring a content ID (Identity), and storing the content ID into a first cache; based on the content ID in the first cache, obtaining a content vector corresponding to the content ID and storing the content vector in a second cache; an index is compiled based on the content ID in the first cache and the content vector in the second cache. The invention further relates to an index construction device, equipment, a medium and a computer program product.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an index construction method, device, equipment, medium and computer program product. Background Art

[0002] Vector retrieval is to retrieve the K vectors (K-Nearest Neighbor, KNN) that are the nearest neighbors to the query vector in a given vector data set. Since KNN has a large amount of calculation, people will only focus on the approximate nearest neighbor (ANN) problem. In practical applications, in the vector recall stage, the candidate set may be very large, and brute force calculation may be very slow. In order to quickly find the most similar neighbor nodes, the method of replacing speed with accuracy is usually used. The specific approach is to use commonly used construction algorithms, such as HNSW, Faiss, and ScaNN, to build an index for the vector, and then find the most similar neighbor nodes from the index. This approach is much faster than brute force calculation, and the accuracy can also meet business requirements. Even so, when the candidate set is very large (for example, in the millions), it usually takes a long time to build the index. Summary of the invention

[0003] The purpose of the present invention is to provide an index construction method, device, equipment, medium and computer program product to solve the technical problem of slow index construction speed.

[0004] A first embodiment of the present invention discloses a method for constructing an index, which is used in an electronic device. The method comprises the following steps:

[0005] Obtaining a content ID, and storing the content ID in a first cache;

[0006] Based on the content ID in the first cache, obtaining a content vector corresponding to the content ID and storing it in a second cache;

[0007] An index is compiled based on the content ID in the first cache and the content vector in the second cache.

[0008] Optionally, the obtaining of the content ID and storing the content ID in the first cache comprises the following steps:

[0009] Obtain the content ID and store it in the third cache;

[0010] The content ID in the third cache is filtered, and the filtered content ID is stored in the first cache.

[0011] According to the method of claim 2, the step of obtaining the content ID and storing it in the third cache comprises the following steps:

[0012] Reading a predetermined number of stored full content IDs every first predetermined time period and storing them in the third cache;

[0013] The real-time content ID uploaded in real time is obtained and stored in the third cache.

[0014] According to the method of claim 3, filtering the content ID in the third cache and storing the filtered content ID in the first cache comprises the following steps:

[0015] filtering the full content IDs in the third cache at second predetermined intervals, and storing the filtered full content IDs in the first cache;

[0016] The real-time content ID in the third cache is filtered, and the filtered real-time content ID is stored in the first cache.

[0017] According to the method of claim 4, the step of obtaining a content vector corresponding to the content ID based on the content ID in the first cache and storing the content vector in the second cache comprises the following steps:

[0018] every third predetermined time period, based on the full content ID in the first cache, obtaining a full content vector corresponding to the full content ID and storing it in the second cache;

[0019] Based on the real-time content ID in the first cache, a real-time content vector corresponding to the real-time content ID is obtained and stored in the second cache.

[0020] Optionally, compiling an index based on the content ID in the first cache and the content vector in the second cache includes:

[0021] dividing the content ID into a plurality of groups;

[0022] Indexes are compiled based on a plurality of sets of the content IDs and the corresponding content vectors in parallel.

[0023] Optionally, the acquiring the content vector corresponding to the content ID and storing it in the second cache further includes:

[0024] Inputting the content ID into a preset vector estimation model to obtain the corresponding content vector;

[0025] Persistently storing the content vector;

[0026] When the content vector in the second cache needs to be restored, the content vector in persistent storage is read and synchronized to the second cache.

[0027] A second embodiment of the present invention discloses an index construction device, comprising:

[0028] A content acquisition module, used to acquire a content ID and store the content ID in a first cache;

[0029] A vector acquisition module, configured to read the content ID in the first cache, acquire the content vector corresponding to the content ID and store it in the second cache;

[0030] The compiling index module is used to read the content ID in the first cache and the content vector in the second cache, and compile an index based on the content ID and the content vector.

[0031] A third embodiment of the present invention discloses an electronic device, which includes a memory storing computer executable instructions and a processor. When the instructions are executed by the processor, the electronic device implements the index construction method according to the first embodiment of the present invention.

[0032] A fourth embodiment of the present invention discloses a computer storage medium, on which instructions are stored. When the instructions are executed on a computer, the computer executes the index construction method according to the first embodiment of the present invention.

[0033] A fifth embodiment of the present invention discloses a computer program product, comprising computer executable instructions, wherein the instructions are executed by a processor to implement the index construction method according to the first embodiment of the present invention.

[0034] Compared with the prior art, the main differences and effects of the embodiments of the present invention are:

[0035] In the present invention, since each step or module is executed in a pipeline manner, the time for serial content acquisition (candidate set acquisition), content filtering, and content vector acquisition is saved, and the group parallel compilation index method is adopted, which greatly shortens the time for building the index (for example, the full link time of 100 minutes can be compressed to 5 minutes); content acquisition (candidate set acquisition), content filtering, and content vector acquisition are divided into two parts, one part processes the full amount of data, and the other part processes the real-time data, that is, the real-time stream of content upload is connected, so the full amount and non-delayed content vectors are always maintained, thereby improving effectiveness.

[0036] In the present invention, the content vector is persistently stored after being obtained. When the service is restarted due to an exception, there is no need to call the vector estimation service again, which greatly shortens the service restart time (for example, the service restart time of 100 minutes can be optimized to 25 minutes) and improves stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A flow chart showing an example index build.

[0038] Figure 2 A flowchart of a method for constructing an index according to an embodiment of the present application is shown.

[0039] Figure 3 A schematic diagram of a system involved in a method for constructing an index according to an embodiment of the present application is shown.

[0040] Figure 4 A flowchart of compiling indexes according to an embodiment of the present application is shown.

[0041] Figure 5 A block diagram of an index construction device according to an embodiment of the present application is shown.

[0042] Figure 6 A hardware structure block diagram of an electronic device that implements the construction of an index according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0043] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0044] In the process of recommendation recall, vector-based approximate nearest neighbor recall is an important part of capturing user interests. It usually requires deep learning-based methods to map users and content into vectors, and then build the content into an index based on the content vector. In this application, content can refer to multimedia notes uploaded by users to the server. After receiving the user request, the most similar content is retrieved from the index based on the user vector. The effectiveness of index construction plays a vital role in capturing user interests and content distribution. Figure 1 As shown, the process of building an index for an example includes the following serial steps:

[0045] S102, obtaining a content candidate set. For example, for tens of millions of contents, it takes about 10 minutes to query and transmit the content candidate set from the database;

[0046] S104, filtering content. For example, after obtaining the content candidate set, filtering is required according to the content positive ranking information, such as pornographic filtering, political filtering, and ecological filtering. This filtering step takes about 10 minutes;

[0047] S106, obtaining a content vector, for example, calling a remote vector estimation service to obtain a content vector after content filtering. This step usually takes about 60 minutes;

[0048] S108, compiling the index, for example, after obtaining the content vector, compiling the index using a commonly used construction algorithm, and the index compilation step usually takes about 20 minutes.

[0049] It can be seen that it takes about 100 minutes to execute the entire link of S102-S108 in series. This index construction method has the following three problems:

[0050] 1) The index building speed is too slow, so the content uploaded in the last 100 minutes cannot be included in the index;

[0051] 2) Content vectors are generated by upstream models and are updated in minutes. However, because the index building speed is too slow, the latest vectors exported by the model cannot be indexed in time.

[0052] 3) When the service crashes and restarts due to an abnormality, the above process steps need to be executed again. The restart time also takes about 100 minutes, resulting in service unavailability within 100 minutes.

[0053] In view of the above problems, the embodiments of the present application provide a preferred index construction method. Figure 2 A flow chart showing the method is shown, Figure 3 A schematic diagram of the system involved in the method is shown below. Figure 2 and Figure 3 Describe it in detail. Figure 2 As shown, the above method includes:

[0054] S202, content acquisition step, acquiring a content ID, and storing the content ID in a first cache; for example, Figure 3 In the scenario shown, the user uploads content to the server on the client, such as graphic notes, audio, video, etc. The ID of the uploaded content is written into the database 301 on the one hand, and connected to the content ID real-time stream 302 on the other hand. The content acquisition module 303 extracts the content ID from the database 301 and the content ID real-time stream 302 as a content candidate set. Subsequently, these extracted content IDs are written into the server memory block, i.e., the cache 307 (first cache).

[0055] According to some embodiments of the present application, the acquisition of content ID and storage of the content ID in the first cache includes the following steps: acquiring content ID and storing it in a third cache; and filtering step: filtering the content ID in the third cache and storing the filtered content ID in the first cache. For example, after the content acquisition module 303 retrieves the content ID from the database 301 and the content ID real-time stream 302, the content ID is written into the server memory block, i.e., cache 306 (third cache). The filtering module 304 retrieves the content ID from cache 306 and filters it according to the positive ranking (there are already tags such as pornography and politics), and writes the filtered content ID into cache 307.

[0056] According to some embodiments of the present application, the content ID is obtained and stored in the third cache, including the following steps: reading a predetermined number of stored full content IDs every first predetermined time period and storing them in the third cache; obtaining the real-time uploaded real-time content ID and storing them in the third cache. For example, the content acquisition module 303 is divided into two parts: full content acquisition 303a and real-time content acquisition 303b. Among them, the full content acquisition 303a scans the database 301 once every first predetermined time period, such as one day, and reads a predetermined number of stored full content IDs. For example, each time the content of nearly one month is scanned, such as 70 million, the content scanned twice may be repeated, and the repeated parts are overwritten; after each scan is completed, the scanned full content ID is put into the cache 306 at one time. The downstream module needs to wait for the full content ID to be put in before it can read these full content IDs, so there is a certain unavailable time (for example, about ten minutes) when it is started for the first time. The real-time content acquisition 303b receives the content ID real-time stream 302, obtains the real-time content ID uploaded by the user in real time, and stores it in the cache 306. If only full scan is performed, even if the full scan is repeated continuously, the content uploaded during this period will not be available because each scan takes a certain amount of time. Therefore, connecting to the real-time stream can ensure that the latest content is obtained without delay.

[0057] According to some embodiments of the present application, the filtering of the content ID in the third cache and storing the filtered content ID in the first cache includes the following steps: filtering the full content ID in the third cache every second predetermined time period and storing the filtered full content ID in the first cache; filtering the real-time content ID in the third cache and storing the filtered real-time content ID in the first cache. For example, the filtering module 304 is divided into two parts: full filtering 304a and real-time filtering 304b. Among them, the full filtering 304a reads the full content ID stored in 303a from the cache 306 every second predetermined time period, for example, one day, and then filters the read full content ID and writes the filtered full content ID into the cache 307; the real-time filtering 304b reads the real-time content ID stored in 303b from the cache 306 in real time, and then filters the read real-time content ID and writes the filtered real-time content ID into the cache 307. Because the content acquisition module 303 is connected to the real-time stream of content upload, the cache 307 always maintains the full content ID without delay.

[0058] S204, vector acquisition step, based on the content ID in the first cache, the content vector corresponding to the content ID is acquired and stored in the second cache; for example, the vector acquisition module 305 takes out the filtered content ID from the cache 307, and sends the content ID to the remote vector estimation service 309. After receiving the content ID, the remote vector estimation service 309 outputs the corresponding content vector and then returns the content vector to the vector acquisition module 305. The vector acquisition module 305 writes the acquired content vector into the server memory block, i.e., the cache 308.

[0059] According to some embodiments of the present application, the obtaining of the content vector corresponding to the content ID based on the content ID in the first cache and storing it in the second cache includes the following steps: obtaining the full content vector corresponding to the full content ID based on the full content ID in the first cache and storing it in the second cache every third predetermined time period; obtaining the real-time content vector corresponding to the real-time content ID based on the real-time content ID in the first cache and storing it in the second cache. For example, the vector obtaining module 305 is divided into two parts: full vector obtaining 305a and real-time vector obtaining 305b. Among them, the full vector acquisition 305a reads the full content ID stored in the full filter 304a from the cache 307 every third predetermined time period, for example, one day, and then sends the read full content ID to the remote vector estimation service 309, and writes the full content vector returned by the remote vector estimation service 309 into the cache 308; the real-time vector acquisition 305b reads the real-time content ID stored in the real-time filter 304b from the cache 307 in real time, and then sends the read real-time content ID to the remote vector estimation service 309, and writes the real-time content vector returned by the remote vector estimation service 309 into the cache 308. Because the content acquisition module 303 is connected to the real-time stream of content upload, the cache 308 always maintains the full and non-delayed content vector.

[0060] According to some embodiments of the present application, the process of obtaining the content vector corresponding to the content ID and storing it in the second cache also includes: inputting the content ID into a preset vector estimation model to obtain the corresponding content vector; persistently storing the content vector; and when the content vector in the second cache needs to be restored, reading the persistently stored content vector and synchronizing it to the second cache. For example, the remote vector estimation service 309 includes a preset vector estimation model, and the vector acquisition module 305 persistently stores the content ID read from the cache 307 to the RocksDB component and sends it to the remote vector estimation service 309. The RocksDB component persists the content vector and synchronizes the content vector to the local disk in real time. When the service is restarted, the content vector in RocksDB can be synchronized to the cache 308 at an extremely fast speed. At this time, there is no need to call the vector estimation model to obtain the content vector, which significantly improves the restart speed of the service and increases the stability and effectiveness of the service. Figure 1 The service restart time of the index building process in the example shown is 100 minutes. By using the RocksDB component to persist the content vector, the service restart time can be optimized to 25 minutes.

[0061] S206, compile index step, compile index based on the content ID in the first cache and the content vector in the second cache. For example, the compile index module 310 cyclically retrieves the content ID from the cache 307, retrieves the corresponding content vector from the cache 308, and then calls the ScaNN algorithm to compile an approximate nearest neighbor index based on the content ID and the content vector, and writes the obtained index into the index real-time stream 311 for vector recall service.

[0062] In the present application, the index may be an approximate nearest neighbor index or any other suitable index.

[0063] In this application, the content acquisition step, filtering step, vector acquisition step and compilation indexing step are decoupled, and threads can be opened separately to execute in a pipeline manner. Correspondingly, the content acquisition module, filtering module, vector acquisition module and compilation indexing module are also decoupled, and threads can be opened separately to execute in a pipeline manner. Since the time for serial content acquisition (candidate set acquisition), content filtering and content vector acquisition is omitted, compared with Figure 1 The entire process of index building in the example shown takes 100 minutes. The entire process can be reduced to 20 minutes by executing each step or module in a pipeline manner.

[0064] According to some embodiments of the present application, compiling the index based on the content ID in the first cache and the content vector in the second cache includes: dividing the content ID into multiple groups; and compiling the index based on the multiple groups of content IDs and the corresponding content vectors in parallel. Figure 4 A flowchart of compiling an index is shown, including:

[0065] S402, reading the content ID from the cache, for example, the compiling index module 310 takes the content ID from the cache 307;

[0066] S404, grouping the content IDs, for example, the compiling index module 310 divides the content IDs into multiple groups, for example, 5 groups;

[0067] S406, reading the content vector from the cache, for example, the compiling index module 310 takes out the corresponding content vector from the cache 308;

[0068] S408, compiling indexes in parallel, for example, the compiling index module 310 independently compiles an approximate nearest neighbor index for each set of content IDs and corresponding content vectors using the ScaNN algorithm;

[0069] S410, writing the index into the index real-time stream, for example, the compiled index module 310 writes the compiled approximate nearest neighbor index into the index real-time stream 311 for vector recall service. S402-S410 are executed in a loop.

[0070] This embodiment adopts the method of grouping and parallel compiling indexes. Figure 1 The full-link time consumption of the index building process in the example shown is 100 minutes, which reduces the full-link time consumption of index building from 100 minutes to 5 minutes. Figure 1 In the index building process of the example shown, new content can only be indexed after 100 minutes of uploading. This embodiment is optimized to allow new content to be indexed after 5 minutes of uploading. Figure 1 In the index building process of the example shown, a vector can enter the index only after 100 minutes of update. This embodiment is optimized to enter the index only after 5 minutes of update.

[0071] Figure 5 A block diagram of an index construction device 500 according to an embodiment of the present application is shown. The device 500 includes:

[0072] Content acquisition module 501, used to acquire a content ID and store the content ID into a first cache;

[0073] A vector acquisition module 502 is used to read the content ID in the first cache, obtain the content vector corresponding to the content ID and store it in the second cache;

[0074] The index compiling module 503 is configured to read the content ID in the first cache and the content vector in the second cache, and compile an index based on the content ID and the content vector.

[0075] The method embodiment of the present application corresponds to this embodiment, and this embodiment can be implemented in conjunction with the method embodiment of the present application. The relevant technical details mentioned in the method embodiment of the present application are still valid in the present embodiment, and are not repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the method embodiment of the present application.

[0076] Figure 6 It is a hardware structure block diagram of an electronic device that implements the construction of an index according to an embodiment of the present application.

[0077] like Figure 6 As shown, the electronic device 600 may include one or more processors 602, a system motherboard 608 connected to at least one of the processors 602, a system memory 604 connected to the system motherboard 608, a non-volatile memory (NVM) 606 connected to the system motherboard 608, and a network interface 610 connected to the system motherboard 608.

[0078] The processor 602 may include one or more single-core or multi-core processors. The processor 602 may include any combination of a general-purpose processor and a special-purpose processor (e.g., a graphics processor, an application processor, a baseband processor, etc.). In an embodiment of the present invention, the processor 602 may be configured to execute the following operations: Figure 2-Figure 4 One or more of the various embodiments shown.

[0079] In some embodiments, system board 608 may include any suitable interface controller to provide any suitable interface to at least one of processors 602 and / or any suitable device or component in communication with system board 608 .

[0080] In some embodiments, the system board 608 may include one or more memory controllers to provide an interface to the system memory 604. The system memory 604 may be used to load and store data and / or instructions. In some embodiments, the system memory 604 of the electronic device 600 may include any suitable volatile memory, such as a suitable dynamic random access memory (DRAM).

[0081] NVM 606 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, NVM 606 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as at least one of a HDD (Hard Disk Drive), a CD (CompactDisc) drive, and a DVD (Digital Versatile Disc) drive.

[0082] NVM 606 may include a portion of storage resources installed on a device of electronic device 600, or it may be accessible by the device but not necessarily a portion of the device. For example, NVM 606 may be accessed over a network via network interface 610.

[0083] In particular, system memory 604 and NVM 606 may include, respectively, a temporary copy and a permanent copy of instruction 620. Instruction 620 may include instructions that, when executed by at least one of processors 602, cause electronic device 600 to implement the following: Figure 2-Figure 4 In some embodiments, instructions 620, hardware, firmware and / or software components thereof may additionally / alternatively be located in system board 608, network interface 610 and / or processor 602.

[0084] The network interface 610 may include a transceiver for providing a radio interface for the electronic device 600, thereby communicating with any other suitable device (e.g., a front-end module, an antenna, etc.) through one or more networks. In some embodiments, the network interface 610 may be integrated with other components of the electronic device 600. For example, the network interface 610 may be integrated with at least one of the processor 602, the system memory 604, the NVM 606, and a firmware device (not shown) having instructions. When at least one of the processors 602 executes the instructions, the electronic device 600 implements Figure 2-Figure 4 One or more of the various embodiments shown.

[0085] The network interface 610 may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface. For example, the network interface 610 may be a network adapter, a wireless network adapter, a telephone modem, and / or a wireless modem.

[0086] In one embodiment, at least one of the processors 602 may be packaged together with one or more controllers for the system motherboard 608 to form a system in package (SiP). In one embodiment, at least one of the processors 602 may be integrated on the same die with one or more controllers for the system motherboard 608 to form a system on a chip (SoC).

[0087] The electronic device 600 may further include an input / output (I / O) device 612 connected to the system motherboard 608. The I / O device 612 may include a user interface to enable a user to interact with the electronic device 600; the design of the peripheral component interface enables the peripheral components to interact with the electronic device 600. In some embodiments, the electronic device 600 further includes a sensor for determining at least one of an environmental condition and location information related to the electronic device 600.

[0088] In some embodiments, I / O device 612 may include, but is not limited to, a display (e.g., a liquid crystal display, a touch screen display, etc.), a speaker, a microphone, one or more cameras (e.g., a still image camera and / or a video camera), a flashlight (e.g., an LED flash), and a keyboard.

[0089] In some embodiments, the peripheral component interface may include, but is not limited to, a non-volatile memory port, an audio jack, and a power interface.

[0090] In some embodiments, the sensors may include, but are not limited to, gyroscope sensors, accelerometers, proximity sensors, ambient light sensors, and positioning units. The positioning unit may also be part of or interact with the network interface 610 to communicate with components of a positioning network (e.g., global positioning system (GPS) satellites).

[0091] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 600. In other embodiments of the present application, the electronic device 600 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0092] Program code can be applied to input instructions to perform the functions described in the present invention and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a system for processing instructions including processor 602 includes any system having a processor such as a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0093] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in the present invention is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0094] One or more aspects of at least one embodiment may be implemented by instructions stored on a computer-readable storage medium, which, when read and executed by a processor, enables an electronic device to implement the method of the embodiment described in the present invention.

[0095] According to some embodiments of the present application, a computer storage medium is disclosed, on which instructions are stored. When the instructions are executed on a computer, the computer executes the index construction method according to the embodiments of the present application.

[0096] The method embodiment of the present application corresponds to this embodiment, and this embodiment can be implemented in conjunction with the method embodiment of the present application. The relevant technical details mentioned in the method embodiment of the present application are still valid in the present embodiment, and are not repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the method embodiment of the present application.

[0097] According to some embodiments of the present application, a computer program product is disclosed, including computer executable instructions, which are executed by a processor to implement an index construction method according to an embodiment of the present application.

[0098] The method embodiment of the present application corresponds to this embodiment, and this embodiment can be implemented in conjunction with the method embodiment of the present application. The relevant technical details mentioned in the method embodiment of the present application are still valid in the present embodiment, and are not repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the method embodiment of the present application.

[0099] It is to be understood that the specific embodiments described herein are only for explaining the present application, rather than limiting the present application. In addition, for ease of description, only some but not all structures or processes related to the present application are shown in the accompanying drawings. It should be noted that in this specification, similar reference numerals and letters represent similar items in the accompanying drawings.

[0100] It should be understood that although the terms "first", "second", etc. may be used herein to describe various features, these features should not be limited by these terms. These terms are used only to distinguish and should not be understood as indicating or implying relative importance. For example, a first feature may be referred to as a second feature, and similarly a second feature may be referred to as a first feature without departing from the scope of the exemplary embodiments.

[0101] In the description of this application, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in this embodiment can be understood according to specific circumstances.

[0102] Illustrative embodiments of the present application include, but are not limited to, methods, devices, apparatuses, media, and computer program products for constructing an index.

[0103] The various aspects of the illustrative embodiments will be described using terms commonly used by those skilled in the art to convey the essence of their work to other persons skilled in the art. However, it is apparent to those skilled in the art that some alternative embodiments may be implemented using the features described in part. For purposes of explanation, specific numbers and configurations are set forth to provide a more thorough understanding of the illustrative embodiments. However, it is apparent to those skilled in the art that alternative embodiments may be implemented without the specific details. In some other cases, some well-known features are omitted or simplified herein to avoid obscuring the illustrative embodiments of the present application.

[0104] In addition, various operations will be described as multiple separate operations in a manner that is most helpful for understanding the illustrative embodiments; however, the order of description should not be interpreted as implying that these operations must rely on the order of description, and many of the operations can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the operations can also be rearranged. When the described operations are completed, the process can be terminated, but can also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0105] References in the specification to "one embodiment," "an embodiment," "an illustrative embodiment," etc., indicate that the described embodiment may include a particular feature, structure, or property, but each embodiment may or may not necessarily include the particular feature, structure, or property. Moreover, these phrases are not necessarily referring to the same embodiment. In addition, when particular features are described in conjunction with a specific embodiment, the knowledge of those skilled in the art can affect the combination of these features with other embodiments, whether or not these embodiments are explicitly described.

[0106] Unless the context dictates otherwise, the terms "comprising," "having," and "including" are synonymous. The phrase "A and / or B" means "(A), (B), or (A and B)."

[0107] As used herein, the term "module" may refer to, be part of, or include: memory (shared, dedicated, or group) for running one or more software or firmware programs, application-specific integrated circuits (ASICs), electronic circuits and / or processors (shared, dedicated, or group), combinational logic circuits, and / or other suitable components that provide the described functionality.

[0108] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order is not necessary. Instead, in some embodiments, these features may be described in a manner and / or order different from that shown in the illustrative drawings. In addition, the structural or method features included in a specific figure do not mean that all embodiments need to include such features. In some embodiments, these features may not be included or may be combined with other features.

[0109] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented in the form of instructions or programs carried or stored on one or more transient or non-transient machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors, etc. When the instructions or programs are run by the machine, the machine may perform the aforementioned various methods. For example, the instructions may be distributed via a network or other computer-readable medium. Therefore, the machine-readable medium may include, but is not limited to, any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, such as a floppy disk, an optical disk, a compact disk read-only memory (CD-ROMs), a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electronically erasable programmable read-only memory (EEPROM), a magnetic card or an optical card, or a flash memory or a tangible machine-readable memory for transmitting network information via electrical, optical, acoustic, or other forms of signals (e.g., carrier waves, infrared signals, digital signals, etc.). Accordingly, machine-readable media includes any form of machine-readable media suitable for storing or transmitting electronic instructions or information readable by a machine (eg, a computer).

[0110] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the use of the technical solution of the present application is not limited to the various applications mentioned in the embodiments of the present application. Various structures and modifications can be easily implemented with reference to the technical solution of the present application to achieve the various beneficial effects mentioned herein. Various changes made within the knowledge of ordinary technicians in the field without departing from the purpose of the present application should all fall within the scope of the patent coverage of the present application.

Claims

1. A method for constructing an index, used in an electronic device, characterized in that: The method comprises the following steps: Obtaining a content ID, and storing the content ID in a first cache; Based on the content ID in the first cache, a content vector corresponding to the content ID is obtained and stored in a second cache; and an index is compiled based on the content ID in the first cache and the content vector in the second cache.

2. The method according to claim 1, characterized in that The step of obtaining a content ID and storing the content ID in a first cache comprises the following steps: Obtain the content ID and store it in the third cache; The content ID in the third cache is filtered, and the filtered content ID is stored in the first cache.

3. The method according to claim 2, characterized in that The step of obtaining the content ID and storing it in the third cache includes the following steps: Reading a predetermined number of stored full content IDs every first predetermined time period and storing them in the third cache; The real-time content ID uploaded in real time is obtained and stored in the third cache.

4. The method according to claim 3, characterized in that The filtering of the content ID in the third cache and storing the filtered content ID in the first cache comprises the following steps: filtering the full content IDs in the third cache at second predetermined intervals, and storing the filtered full content IDs in the first cache; The real-time content ID in the third cache is filtered, and the filtered real-time content ID is stored in the first cache.

5. The method according to claim 4, characterized in that The acquiring, based on the content ID in the first cache, the content vector corresponding to the content ID and storing it in the second cache comprises the following steps: every third predetermined time period, based on the full content ID in the first cache, obtaining a full content vector corresponding to the full content ID and storing it in the second cache; Based on the real-time content ID in the first cache, a real-time content vector corresponding to the real-time content ID is obtained and stored in the second cache.

6. The method according to claim 1, characterized in that The compiling an index based on the content ID in the first cache and the content vector in the second cache includes: dividing the content IDs into a plurality of groups; Indexes are compiled based on a plurality of sets of the content IDs and the corresponding content vectors in parallel.

7. The method according to claim 1, characterized in that The step of obtaining the content vector corresponding to the content ID and storing it in the second cache further includes: Inputting the content ID into a preset vector estimation model to obtain the corresponding content vector; Persistently storing the content vector; When the content vector in the second cache needs to be restored, the content vector in persistent storage is read and synchronized to the second cache.

8. An index construction device, characterized in that: include: A content acquisition module, used to acquire a content ID and store the content ID in a first cache; A vector acquisition module, configured to read the content ID in the first cache, acquire the content vector corresponding to the content ID and store it in the second cache; The compiling index module is used to read the content ID in the first cache and the content vector in the second cache, and compile an index based on the content ID and the content vector.

9. An electronic device, characterized in that: The electronic device comprises a memory storing computer executable instructions and a processor. When the instructions are executed by the processor, the electronic device implements the index construction method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that: Instructions are stored on the computer storage medium. When the instructions are executed on a computer, the computer is enabled to execute the index construction method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The method comprises computer executable instructions, wherein the instructions are executed by a processor to implement the index construction method according to any one of claims 1 to 7.

Images