Data processing method, apparatus, device, medium, and product

Abstract

Embodiments of the present application disclose data processing methods, devices, equipment, media and products, which can be applied to the technical field of data processing. The method comprises: obtaining a target word list, and storing an inverted link of each word in the target word list in a first storage area; the target word list is at least one word determined by a data access node to meet a heat condition; finding the inverted link of the target word in a second storage area of a data retrieval node, and when the inverted link of the target word is not found, obtaining a document jump step length that meets a jump constraint condition; based on the storage address of the document identifier in the inverted link of the target word and the document jump step length, performing document jump addressing in a plurality of physical storage units contained in the first storage area, and loading the inverted link of the target word into the second storage area based on the addressed physical storage unit. The embodiments of the present application help to improve the efficiency of data preheating.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to data processing methods, apparatus, equipment, media and products. Background Technology

[0002] Currently, in data query systems, in order to better provide retrieval services, the server can preload relevant data (such as inverted indexes of terms) from the disk into memory (i.e., preheat the server). The current preheating method is usually to re-execute all query requests in the historical query log and load the inverted indexes of each term involved in all query requests into memory.

[0003] However, the inventors discovered during practice that when warming up data by re-executing query requests from historical query logs, a large number of low-frequency words accumulate in memory. These low-frequency words are rarely used in actual retrieval services but consume memory space, and repeatedly executing historical query logs takes considerable time, resulting in low data warming efficiency. Furthermore, when loading the inverted index of each term into memory, each document identifier in the inverted index is typically read sequentially. Since physical pages on disk load all data into memory each time data is read, sequentially reading all document identifiers results in repeated data loading, further reducing the loading efficiency of the inverted index and thus lowering the efficiency of data warming. Therefore, improving the efficiency of data warming is a problem that urgently needs to be solved. Summary of the Invention

[0004] This application provides a data processing method, apparatus, device, medium, and product that can obtain a list of target terms that meet the popularity criteria through a data access node, and use a jump addressing method to move the inverted chain of terms in the target term list from the first storage area to the second storage area, thereby helping to improve the efficiency of data preheating.

[0005] On one hand, embodiments of this application disclose a data processing method, which is applied to a data processing system, including a data access node and a data retrieval node; the method is executed by the data retrieval node, and includes: The target term list is obtained from the data storage object associated with the data access node, and the inverted index of each term in the target term list is stored in the first storage area; the target term list is at least one term that satisfies the popularity condition as determined by the data access node; at least one term includes the target term. The inverted index of the target term is searched in the second storage area of ​​the data retrieval node. If the inverted index of the target term is not found, the document jump step size that satisfies the jump constraint is obtained. The document jump step size refers to the number of document identifiers skipped each time the document identifiers in the inverted index of the target term are read from the first storage area. Based on the storage address of the document identifier in the inverted list of the target term and the document jump step size, document jump addressing is performed in multiple physical storage units contained in the first storage area. Based on the addressed physical storage unit, the inverted list of the target term is loaded into the second storage area to provide data retrieval services based on the inverted list in the second storage area.

[0006] On one hand, embodiments of this application disclose a data processing method, which is applied to a data processing system, including a data access node and a data retrieval node; the method is executed by the data access node and includes: Create multiple threads for term counting; each thread maintains a buffer area; the target thread is included among the multiple threads. When the target thread obtains at least one term parsed from the target query request, it writes the count value of each term in the target query request into the target buffer area maintained by the target thread. When the aggregation condition of the target buffer region is met, the count value of each term in the target buffer region is aggregated to obtain the aggregated count value of at least one candidate term. Write the aggregated count value of at least one candidate term into a global counting structure, statistically obtain at least one term that meets the popularity condition based on the global counting structure, and determine the target term list based on at least one term that meets the popularity condition. The target term list is stored in a data storage object associated with the data access node, so that the data retrieval node can obtain the target term list from the data storage object.

[0007] On one hand, embodiments of this application disclose a data processing apparatus, which is applied to a data processing system, the data processing system including a data access node and a data retrieval node; the apparatus operates at the data retrieval node, and the apparatus includes: The list retrieval unit is used to retrieve a list of target terms from a data storage object associated with the data access node, and to store the inverted index of each term in the target term list in the first storage area; the target term list is at least one term that satisfies the popularity condition as determined by the data access node; at least one term includes the target term; The step size acquisition unit is used to search for the inverted chain of the target term in the second storage area of ​​the data retrieval node, and when the inverted chain of the target term is not found, to obtain the document jump step size that satisfies the jump constraint condition; the document jump step size refers to the number of document identifiers skipped each time the document identifiers in the inverted chain of the target term are read from the first storage area. The skip addressing unit is used to perform document skip addressing in multiple physical storage units contained in the first storage area based on the storage address of the document identifier in the inverted list of the target term and the document skip step size. Based on the addressed physical storage unit, the inverted list of the target term is loaded into the second storage area to provide data retrieval services based on the inverted list in the second storage area.

[0008] On one hand, embodiments of this application disclose a data processing apparatus, which is applied to a data processing system, the data processing system including a data access node and a data retrieval node; the apparatus operates at the data access node, and the apparatus includes: A thread creation unit is used to create multiple threads for term counting; each of the multiple threads maintains a buffer area; the multiple threads include a target thread; The thread counting unit is used to write the count value of each term in the target query request into the target buffer area maintained by the target thread when the target thread obtains at least one term parsed from the target query request. The aggregation counting unit is used to aggregate the count value of each term in the target buffer area when the aggregation condition of the target buffer area is met, so as to obtain the aggregation count value of at least one candidate term. A global counting unit is used to write the aggregated count value of at least one candidate term into a global counting structure, statistically obtain at least one term that meets the popularity condition based on the global counting structure, and determine the target term list based on at least one term that meets the popularity condition. The list storage unit is used to store the list of target terms into a data storage object associated with the data access node, so that the data retrieval node can obtain the list of target terms from the data storage object.

[0009] One embodiment of this application provides a computer device, including a memory and a processor; the memory is connected to the processor, the memory is used to store a computer program, and the processor is used to invoke the computer program so that the computer device performs the method provided in any of the above embodiments.

[0010] One aspect of this application provides a computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor, so that a computer device having a processor performs the method provided in any of the above aspects.

[0011] One aspect of this application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The computer program is adapted to be read and executed by a processor so that a computer device having a processor performs the method provided in any of the above aspects.

[0012] Using the embodiments of this application, a list of target terms can be obtained from a data storage object associated with a data access node, and the inverted index of each term in the target term list is stored in a first storage area. This target term list consists of at least one term determined by the data access node that satisfies a popularity condition, and this at least one term includes the target term. Further, the inverted index of the target term is searched in the second storage area of ​​the data retrieval node. If the inverted index of the target term is not found, a document jump step size satisfying a jump constraint condition is obtained. This document jump step size refers to the number of document identifiers skipped each time a document identifier in the inverted index of the target term is read from the first storage area. Then, based on the storage address of the document identifier in the inverted index of the target term and the document jump step size, document jump addressing is performed in multiple physical storage units included in the first storage area. Based on the addressed physical storage unit, the inverted index of the target term is loaded into the second storage area to provide data retrieval services based on the inverted index in the second storage area. Based on this, a list of target terms that meet the popularity criteria can be obtained through data access nodes, and the inverted chain of terms in the target term list can be moved from the first storage area to the second storage area through jump addressing, which helps to improve the efficiency of data preheating. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application; Figure 2 This is a schematic diagram of a data processing method provided in an embodiment of this application; Figure 3 This is a flowchart illustrating a data processing method provided in an embodiment of this application; Figure 4 This is a schematic diagram illustrating the effect of document skip addressing provided in an embodiment of this application; Figure 5 This is a schematic diagram of a document skip addressing process provided in an embodiment of this application; Figure 6 This is a flowchart illustrating a data processing method provided in an embodiment of this application; Figure 7 This is a schematic diagram illustrating the effect of hot word statistics provided in an embodiment of this application; Figure 8 This is a flowchart illustrating a counting mechanism for counting segments provided in an embodiment of this application; Figure 9 This is a schematic diagram illustrating the effect of a data processing system provided in an embodiment of this application; Figure 10 This is a timing interaction diagram of a data processing method provided in an embodiment of this application; Figure 11 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application; Figure 12 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application; Figure 13 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0016] The following describes some technical terms used in the embodiments of this application: An inverted list (or postings list) is an ordered list of all document identifiers (DocIds, also known as document IDs) containing a given search term. This inverted list can be a core data structure for data retrieval, used to store the occurrence information of a search term within a document collection. Within a search term's inverted list, the documents containing the search term (i.e., all documents containing the search term) can be arranged in ascending order of their document identifiers (DocIds, also known as document IDs). Essentially, it's a pre-built mapping relationship of "search term → list of document IDs containing the term," used to accelerate the retrieval and querying of massive amounts of data. Each term is associated with an ordered list of document IDs, typically expressed as: keyword → [docid_1, docid_2,..., docid_N], where docid_N represents the document ID.

[0017] Document (Document): In a data retrieval scenario, a document that can be searched. In this application, a document can be published content, users, products, web pages, or other data; no limitation is made here.

[0018] DocId (Document Identifier): This is a unique identifier assigned to each document, i.e., a document identifier. DocIds can be used to arrange documents in ascending order in the index to facilitate sequential indexing during document retrieval.

[0019] A page fault is a mechanism in which the processor accesses a memory address and finds that the physical page at that address is not in random access memory, triggering a hardware interrupt. The operating system kernel then loads the corresponding disk data block into physical memory.

[0020] Memory: It is a type of volatile storage. When the computer is powered off, the data stored in memory will be lost immediately.

[0021] Disk: A type of non-volatile storage, the data on a disk can be retained for a long time even when the power is off, making it a permanent storage medium for computers.

[0022] Direct Memory Access (DMI) refers to a hardware mechanism where the disk controller writes data directly to random access memory (RAM) without the processor. In this embodiment, after a page fault is triggered, this mechanism can load physical pages into the operating system's page cache. DMI can utilize a dedicated DMA controller (DMAC) to handle data transfer: the CPU only needs to perform initialization configuration (setting the address, data volume, and transfer direction) before the transfer begins. The entire transfer process is completed independently by the DMA controller, and only an interrupt notification is sent to the CPU upon completion, without CPU involvement. Compared to traditional methods (where the CPU needs to copy data segment by segment from peripherals to memory), DMA avoids CPU occupancy, allowing the CPU to process other tasks in parallel, significantly improving overall system efficiency.

[0023] A thread is the smallest unit of computation that an operating system can schedule. It is contained within a process and is the actual unit of operation of that process. Multiple threads within the same process can share all the system resources of that process, such as virtual address space and file descriptors. At the same time, each thread has its own independent call stack, register environment, and thread-local storage.

[0024] Local Storage Area (TLS): A private storage area for each worker thread, invisible to different threads, and read / write operations do not require locking. This area does not rely on conventional global data segments or thread stack management, thus avoiding synchronization race issues when multiple threads access shared variables.

[0025] Please seeFigure 1 , Figure 1 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application. For example... Figure 1 As shown, the data processing system may include terminal devices (such as device 11a, device 12a, and device 13a) and a service cluster 100a. It is understood that... Figure 1 The number of terminal devices and service clusters shown is merely illustrative; any number of terminal devices and service clusters can be used as needed. Terminal devices (such as devices 11a, 12a, and 13a) can communicate with service cluster 100a via a network (i.e., a medium providing a communication link through wired, wireless communication links, or fiber optic cables, etc.) to transmit data.

[0026] The service cluster 100a can include various types of service nodes for providing services, such as nodes 101a, 102a, and 103a. Node 101a can be a data access node, a service node deployed at the request access layer in the distributed search engine, responsible for receiving user query requests and distributing them to retrieval layer nodes for actual processing. Furthermore, in this application, the data access node can collect hot words online, achieving lock-free streaming statistics through a multi-level buffer aggregation mechanism (the specific method of which can be seen in the following detailed description). It periodically reports the Top-K hot words (i.e., the K most popular hot words) to the event log stream for persistent storage of the hot word list. Hot words refer to terms (also called search terms) that meet the popularity criteria. Popularity can be reflected by the number of times a term appears in various query requests; that is, hot words can be determined by counting the number of times a term appears in each query request.

[0027] Node 102a can be an offline service node (also called an offline processing node) used for offline processing. This offline service node can process the consumption event logs of various service clusters offline, and aggregate global Top-K hot words (i.e., terms that meet the popularity criteria) by service cluster and date to generate a target term list, thereby obtaining a hot word snapshot file (i.e., a snapshot file of the target term list). Furthermore, it can persistently store the data in a data storage object and write a version completion tag to ensure data integrity and consistency during retrieval. The data storage object can be an object used to store offline data in the distributed search engine. This data storage object can provide a persistent storage service shared across nodes, storing snapshot files of hot words (i.e., terms that meet the popularity criteria), and supporting on-demand retrieval by any node.

[0028] Node 103a can be a data retrieval node used to provide retrieval services. It can be a service node deployed in the retrieval execution layer of a distributed search engine, responsible for loading index files and performing actual retrieval calculations. Specifically, after the index is loaded and before external services begin, node 103a can pull the latest hot word snapshot file (i.e., snapshot file of the target term list) from remote storage (i.e., data storage object). Through physical page-aware skip addressing (the specific method can be seen in the following detailed description), it fills the inverted chain corresponding to the hot words from the disk (i.e., the first storage area) to the page cache (i.e., the second storage area), and then opens the service port to receive traffic to provide retrieval services.

[0029] It is understood that a client may run on the terminal device (such as device 12a), which may be a program providing local services to the user (also referred to as a business object or operation object). The service cluster 100a may be the service cluster corresponding to the client, and programs for providing resources, service data, etc., to the client may run within this service cluster 100a. It is understood that the client running on the terminal device may also be called an application client, business client, etc., and a business account may be logged into the client; the currently logged-in account may be called the current account. For example, the client running on the terminal device may be a client for social networking, communication, or data search. In this embodiment, the business object (such as user U1) can interact with the service cluster through the terminal device to perform data queries (also referred to as data retrieval or data search), sending query requests to the service cluster and receiving query results returned by the service cluster based on the query requests.

[0030] It is understood that terminal devices (such as device 11a) may include, but are not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, smart speakers, etc., without limitation. Any service node in the service cluster 100a may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms, without limitation.

[0031] Please see Figure 2 , Figure 2 This is a schematic diagram illustrating a data processing method provided in an embodiment of this application. For example... Figure 2 As shown, this data processing method may involve data access nodes (such as node 101a) and data retrieval nodes (such as node 103a). The data access node can handle query requests (such as...). Figure 2As shown in 201a), the data access node can then parse the query request (such as...). Figure 2 As shown in 202a), at least one term can be obtained, which can be obtained by segmenting the search text in the query request. Furthermore, this can be achieved through a hot word statistics module (such as...). Figure 2 As shown in 203a, at least one term (e.g., as shown in 203a) that satisfies the heat condition is determined. Figure 2 As shown in 206a), for example, at least one term satisfying the popularity condition can be a term ranked among the top K by the count value of terms counted in the current data access node, also known as Top-K hot words. Furthermore, a target term list can be determined based on at least one term satisfying the popularity condition, and then, through data storage objects (such as...) Figure 2 As shown in 207a, this snapshot file stores the target term list. This snapshot file is a structured, persistent file used to record the target term list at a specific statistical point in time. The data storage object can be a remote storage object; that is, it can be a storage object deployed outside the current data access node and accessed remotely via a network. This ensures that even if the data access node experiences problems, the data retrieval node can still obtain the persistently stored target term list through this data storage object.

[0032] In the hot word statistics module, a multi-level buffer aggregation mechanism can be used for word counting. This mechanism employs a "thread-local buffer + in-thread pre-aggregation" approach, decoupling the existing streaming probability counting structure from the high-concurrency online request processing chain. This makes the performance impact of online hot word statistics on the main query chain negligible. Specifically, multiple threads can be created in the hot word statistics module; for example, these threads can be configured as follows: Figure 2 As shown in 204a, it includes: Thread 1, Thread 2, and Thread N. Each thread's Local Storage Area (TLS) can maintain a buffer area (also called a private buffer) to record the terms involved in a single query request and their corresponding counts. Furthermore, when the aggregation condition of a thread (such as Thread 1) is met (e.g., the counts recorded in the buffer area reach a capacity threshold, or the current time reaches a preset aggregation time), the counts in Thread 1's buffer area can be aggregated to obtain an aggregation result. This merges identical terms into one term (i.e., a candidate term), and the counts of identical terms are added together to obtain the aggregate count value corresponding to the candidate term. In other words, the aggregation result can include at least one candidate term and the aggregate count value of each candidate term. Furthermore, the aggregation result can be written to a global counting structure (e.g., ...). Figure 2As shown in 205a), the global count value of all terms involved in the query requests accessed by the current data access node is statistically analyzed, so that at least one term that meets the popularity condition can be determined based on the global count value of each term.

[0033] Furthermore, the data retrieval node can pull a list of target terms from the data storage object, specifically a snapshot file of the target term list. In the data retrieval node, if the inverted index of a certain hot term does not exist in the second storage area, then the inverted index of that hot term can be retrieved from the first storage area (e.g., ...). Figure 2 As shown in 208a, it is loaded into the second storage area (e.g., Figure 2 As shown in 209a), this allows data retrieval nodes to quickly retrieve the inverted index of the hot keyword from the second storage area when providing retrieval services, thus improving retrieval efficiency. Specifically, when loading the inverted index from the first storage area using skip addressing, the skip addressing method establishes a mapping constraint between the "logical read step size" and the "physical storage unit size," triggering the loading of the largest physical storage unit (e.g., a physical page) with the fewest first storage area (e.g., disk) addressings, achieving high-efficiency hot starts. Specifically, when loading the inverted index from the first storage area, the document skip step size (e.g., ...) can be obtained. Figure 2 (as shown in 210a), and then perform document jump addressing in multiple physical storage units in the first storage area based on the document jump step size (e.g. Figure 2 As shown in 211a), the inverted index of each term in the target term list is then loaded from the first storage area to the second storage area (as shown in 211a). Figure 2 As shown in 212a), the document jump step size refers to the number of document identifiers (i.e., the number of document identifiers) skipped each time a document identifier in the inverted list of the target term is read from the first storage area. In other words, the document jump step size can refer to the number of document identifiers skipped when reading the inverted list of the target term from the first storage area. It is understood that the inverted lists of each term in the target term list can be loaded into a second storage area (such as memory) using the above method. For example, the second storage area may include, for example, the inverted lists of each term in the target term list. Figure 3As shown in 213a, the inverted chain 1 corresponding to hot word 1, the inverted chain 2 corresponding to hot word 2, and the inverted chain K corresponding to hot word K, etc., ensure that the inverted chains stored in the second storage area of ​​the data storage node are inverted chains of hot words (i.e., terms that meet the popularity criteria), rather than low-frequency terms occupying memory space, which helps to improve retrieval efficiency during data retrieval. By using a skip addressing method to load the inverted chain from the first storage area to the second storage area, each physical storage unit (such as a physical page) only needs to be read once, and no physical storage units involved in the inverted chain are missed. In addition, in this application, an end-to-end system is connected through a storage-compute separation architecture, that is, the hot word collection end (i.e., data access node) and the data retrieval execution end (i.e., data retrieval node) are decoupled. The hot word list is shared through remote storage (i.e., data storage object), so that any data retrieval node can complete a hot start without local historical data.

[0034] It should be noted that all user data collected in this application is collected with the user's consent and authorization, and the collection, use, and processing of relevant user data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, this application may display prompts before and during the collection of user data to inform the user that their data is being collected. This ensures that the application only begins the steps for collecting user data after receiving confirmation from the user regarding the prompt; otherwise (i.e., without receiving confirmation from the user), the steps for collecting user data are terminated, meaning that user data is not collected.

[0035] It is understood that the above scenarios are merely examples and do not constitute a limitation on the application scenarios of the technical solutions provided in the embodiments of this application. The technical solutions of this application can also be applied to other scenarios. For example, as those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0036] Based on the above description, this application proposes a data processing method. Please refer to [link to relevant documentation]. Figure 3 , Figure 4 This is a flowchart illustrating a data processing method provided in an embodiment of this application. The method is applied to a data processing system, which includes a data access node and a data retrieval node; the method is executed by the data retrieval node. The data processing method may include the following steps S101-S103.

[0037] S101. Obtain a target term list from the data storage object associated with the data access node, and store the inverted index of each term in the target term list in the first storage area; the target term list is at least one term that satisfies the popularity condition determined by the data access node; at least one term includes the target term.

[0038] For details regarding the data access nodes, please refer to the relevant descriptions above; they will not be repeated here.

[0039] The target term list can be a list determined by at least one term that meets the popularity criteria. Specifically, the target term list can be at least one term that meets the popularity criteria, determined by the data access node.

[0040] In this application, at least one term in the target term list includes a target term. The target term can be any term in the target term list. It is understood that in this application, each term in the target term list can be used as a target term. Therefore, the method provided in this application embodiment can be used to add the inverted index of each term in the target term list to the second storage area for pre-warming up the data retrieval node. That is, the terms in the target term list are pre-loaded into the second storage area of ​​the data retrieval node so that the inverted index of the term corresponding to the search text can be obtained more quickly when providing data retrieval services, thus completing the data retrieval.

[0041] The popularity condition refers to the conditions that a term must meet to be included in the target term list in terms of popularity. It can be understood that popularity here can be represented by the query frequency of the term, which refers to the frequency with which the term appears in query requests within the data retrieval system.

[0042] Optionally, the popularity condition can be that the query frequency of a term ranks among the top K in the total query frequency of all terms in the data retrieval system, where K is a positive integer. Alternatively, the popularity condition can be that the query frequency of a term is greater than or equal to a preset value. Or, the popularity condition can be that the query frequency of a term ranks among the top K in the total query frequency of all terms in the data retrieval system, and the query frequency of the term is greater than or equal to a preset value.

[0043] Understandably, for ease of description, the terms in the target term list can be referred to as hot terms. The specific method by which the data access node determines at least one term in the target term list can be found in the subsequent related descriptions, which will not be elaborated here.

[0044] The first storage area can be a storage area in the data retrieval node used for long-term data storage. For example, the first storage area can be the disk of the data retrieval node.

[0045] It is understood that the first storage area may include multiple physical storage units. A physical storage unit can be the smallest hardware unit in which the operating system of the data retrieval node manages the reading and writing of the first storage area (such as a disk). For example, the physical storage unit can be a physical page.

[0046] Each physical storage unit can have a corresponding unit storage capacity, which can be the amount of data that a physical storage unit can store. In other words, each physical storage unit can be used to store a certain amount of data (i.e., the unit storage capacity). For example, the unit storage capacity of the physical storage unit can be 4096 bytes.

[0047] S102. Search for the inverted list of the target term in the second storage area of ​​the data retrieval node, and if the inverted list of the target term is not found, obtain the document jump step size that satisfies the jump constraint condition; the document jump step size refers to the number of document identifiers skipped each time the document identifiers in the inverted list of the target term are read from the first storage area.

[0048] The second storage area can be a storage area in the data retrieval node used for temporary data storage. For example, the second storage area can be the memory of the data retrieval node.

[0049] Understandably, if the inverted index of the target term is not found in the first storage area, it can be obtained from the second storage area of ​​the data retrieval node. Specifically, the inverted index of the target term can be obtained by performing jump addressing from the first storage area based on the document jump step size, so as to load the inverted index of the target term from the first storage area to the second storage area.

[0050] Understandably, the smallest hardware unit in the data retrieval node's operating system that manages read and write operations in the first storage area (such as a disk) can be called a physical storage unit, for example, a physical page (4096 bytes). When the data retrieval node's processor executes a memory addressing instruction, if the target physical page (i.e., the physical storage unit) does not exist in the second storage area (such as memory), meaning the data to be read is not in the second storage area, the hardware automatically triggers a page fault. The operating system kernel then loads all 4096 bytes of the target physical page into the page cache at once via direct memory access. Subsequently, when data in the target physical page needs to be used, it can be retrieved directly from memory without needing to access the slow first storage area (such as a disk).

[0051] Understandably, when reading data from a physical page in the first storage area, the application only needs to logically "touch" any single byte within a physical storage unit (using a physical page as an example). Then, all the data in the physical page (e.g., 4096 bytes) is automatically loaded into the page cache (i.e., the second storage area). However, when reading the inverted index of a term, all document IDs involved in the inverted index are read sequentially. Since a physical page may contain multiple document IDs in an inverted index, sequentially reading all document IDs involves repeatedly loading the physical page data into memory, resulting in significant data read redundancy and extremely low efficiency in loading the term's inverted index into the second storage area. Therefore, this application can load the inverted index from the first storage area to the second storage area using a skip addressing method when reading the term's inverted index from the first storage area. This ensures that each physical page only needs to be read once, and no physical pages involved in the inverted index are missed.

[0052] The document jump step size refers to the number of document identifiers (i.e., the number of document identifiers) skipped each time a document identifier in the inverted list of a target term is read from the first storage area. In other words, the document jump step size can refer to the number of document identifiers skipped when reading the inverted list of a target term from the first storage area. In other words, the document jump step size can be the number of documents between the previous and subsequent document identifier readings. For example, if the document jump step size is 10, and the first document identifier read is the 1st document identifier in the inverted list, the second document identifier read is the 11th document identifier in the inverted list, and the third document identifier read is the 21st document identifier in the inverted list.

[0053] The jump constraint can be a constraint that the document jump step size must satisfy. This constraint ensures that when reading the inverted index, each physical page only needs to be read once, and no physical pages involved in the inverted index are missed. In other words, the jump constraint means that the disk space spanned by each jump document identifier is less than the size of a physical page, ensuring that each jump falls within a new physical page not yet covered by the previous address, without missing any physical pages. Optionally, the jump constraint can be that the document jump step size is less than or equal to a reference value, which can be determined based on the storage size of each physical storage unit (such as a physical page) and the average storage data size of document identifiers in the inverted index.

[0054] In one possible implementation, obtaining the document jump step size that satisfies the jump constraint may include the following steps: 1. Obtaining the unit storage size of a single physical storage unit and determining the average identifier data size based on the storage data size of the document identifier in the inverted list; 2. Calculating a reference value based on the unit storage size and the average identifier data size; 3. Constructing jump constraints based on the reference value; the jump constraint is that the document jump step size is less than the reference value; 4. Determining the document jump step size that satisfies the jump constraint from the values ​​that are less than the reference value.

[0055] For details regarding the storage capacity of a single unit, please refer to the relevant descriptions above; they will not be repeated here.

[0056] The average identifier data size can refer to the average storage data size of document identifiers in the inverted index. Optionally, the average identifier data size can be the average storage data size of document identifiers in the inverted index of the target term, or it can be the average storage data size of document identifiers in the inverted index of all terms in the target term list; this is not limited here. The storage data size of a document identifier can refer to the number of bytes corresponding to a document identifier during storage; for example, the storage data size of a document identifier can be 4 bytes. It is understood that storing a document identifier can be done by directly storing the document identifier itself, in which case the storage data size of the document identifier can be the number of bytes of the document identifier itself; or, storing a identifier encoding obtained by encoding the document identifier (such as difference encoding), in which case the storage data size of the document identifier can be the number of bytes of the identifier encoding corresponding to the document identifier. It is understood that storing document identifiers after encoding reduces the amount of storage data required for the document identifier, thereby reducing the storage space occupied by the inverted index and lowering storage costs.

[0057] Optionally, the average identifier data size can be calculated based on the total storage data of all document identifiers in the inverted index (such as the inverted index of the target term, or the inverted index of all terms in the target term list); or, it can be calculated based on the storage data of a preset number of document identifiers extracted from the inverted index. Therefore, it is not necessary to obtain the storage data of all document identifiers in the inverted index to calculate the average identifier data size; instead, a portion of the document identifiers can be extracted to calculate the average document identifier data size, reducing the computational load and improving the efficiency of the average identifier data size calculation. When extracting document identifiers for calculating the average identifier data size, they can be randomly selected or extracted at certain document intervals (such as one identifier extracted every 200 document identifiers); this is not limited here.

[0058] The reference value can be data used to determine the document jump step size, and this reference value can be calculated based on the unit storage size and the average identifier data size. For example, the reference value can be the unit storage size divided by the average identifier data size.

[0059] The jump constraint can be determined based on a reference value. For example, the jump constraint is that the document jump step size is less than the reference value. Furthermore, the document jump step size that satisfies the jump constraint can be determined from values ​​that are less than the reference value.

[0060] For example, the jump constraint can be expressed by the following formula (1): Formula (1) Where Y represents the document jump step size, P is the unit storage size of the physical storage unit, and D represents the average size of the inverted index identifier data. This represents a reference value calculated based on the unit storage capacity and the average amount of identifier data.

[0061] Understandably, when determining the document jump step size that satisfies the jump constraint from values ​​smaller than the reference value, the document jump step size should not be too large or too small, but rather a suitable value should be chosen.

[0062] Optionally, in this embodiment, a suitable value for the document jump step size can be determined through a prior comparative experiment. Here, we take a unit storage size of 4096 bytes and an average identifier data size of 4 as an example. The results obtained when taking different stable jump step sizes can be seen in Table (1) below.

[0063] Table (1)

[0064] Based on the above description, it can be seen that a document jump step size of 128 is most suitable. The selection is based on the following: under the premise of satisfying the jump constraint, Y=128 (i.e., the document jump step size is 128) controls the amount of redundant copying in a single read to 512 bytes (about 1 / 8 of the physical page), which reduces the number of disk addressing times by 87.5% compared to Y=1, and retains a sufficient safety margin compared to the critical value Y=1024 (this is because the actual number of bytes obtained by differential encoding of the document identifier fluctuates, and a conservative 1 / 8 is taken as the safety factor).

[0065] S103. Based on the storage address of the document identifier in the inverted list of the target term and the document jump step size, perform document jump addressing in multiple physical storage units contained in the first storage area, and load the inverted list of the target term into the second storage area based on the addressed physical storage unit, so as to provide data retrieval services based on the inverted list in the second storage area.

[0066] The storage address refers to the address where a document identifier (such as the document identifier of the first document identifier) ​​in the inverted chain is stored in a storage area (such as the first storage area). This storage address indicates the physical storage unit (such as a physical page) where the document identifier (such as the first document identifier) ​​resides and its offset within that physical storage unit. Each physical page is addressed starting from a beginning address, and the offset is the difference between the data (such as the first document identifier) ​​and the beginning address of this physical page.

[0067] Understandably, when the data retrieval node is started, the index file can be loaded in the second storage area. This index file can record the storage address of the document identifier in the inverted chain of each term in the target term list. Then, the storage address of the document identifier in the inverted chain of the target term can be determined from the index file, so that when a page fault is triggered, the inverted chain can be loaded from the first storage area to the second storage area based on the storage address of the document identifier.

[0068] In this context, document skipping addressing can refer to skipping some document identifiers when reading document identifiers in the inverted chain from the first storage area, based on the document skipping step size. However, since the above application only needs to logically "touch" any byte in a certain physical storage unit (taking a physical page as an example), then all the data of the physical page (such as 4096 bytes of data) will be automatically loaded into the page cache (i.e., the second storage area). Even if only some document identifiers in the inverted chain are read, the final result can still pull all document identifiers in the inverted chain into the second storage area.

[0069] In one possible implementation, based on the storage address of the document identifier in the inverted list of the target term and the document jump step size, document jump addressing is performed in multiple physical storage units contained in the first storage area. The inverted list of the target term is then loaded into the second storage area based on the addressed physical storage unit. This may include the following steps: 1. Obtaining the storage address of the first document identifier in the inverted list of the target term in the first storage area; when reading the first document identifier based on its storage address, loading all data in the physical storage unit where the first document identifier is located into the second storage area; the first document identifier is the first document in the inverted list of the target term. 1. Identify; 2. Determine the second document identifier to be read in the inverted list of the target term based on the document jump step and the first document identifier; the number of document identifiers that differ between the second document identifier and the first document identifier is the document jump step; 3. Obtain the storage address of the second document identifier in the first storage area. When reading the second document identifier based on the storage address of the second document identifier, load all data in the physical storage unit where the second document identifier is located into the second storage area. If the inverted list of the target term is not fully loaded into the second storage area, update the second document identifier based on the document jump step to load the inverted list of the target term into the second storage area.

[0070] The first document identifier is the first document identifier in the inverted list of the target term. For example, if the inverted list of term C1 is [D1-D2-D3....D300], then the first document identifier can be D1.

[0071] It is understandable that when reading the first document identifier based on the storage address of the first document identifier, all the data in the physical storage unit where the first document identifier is located can be loaded into the second storage area, that is, loaded into the operating system page cache.

[0072] The second document identifier can be the next document identifier to be read in the inverted list of the target term. This second document identifier can be determined based on the document jump step and the first document identifier. The number of document identifiers between the second document identifier and the previously read document identifier (such as the first document identifier) ​​is the document jump step. It can be understood that after reading the first document identifier, the document identifier following the first document identifier in the inverted list is not read directly, but rather the document identifier separated from the first document identifier by a document jump step, which is the so-called skip reading. Specifically, based on the document order of the first document identifier in the inverted list and the document jump step, the target document order of the next document identifier to be read in the inverted list can be determined, and the document identifier corresponding to the target document order in the inverted list is used as the second document identifier. For example, the inverted index of term C1 is [D1-D2-D3....D300], and the first document identifier is the first document identifier in the inverted index, that is, the first document identifier is D1. If the document jump step is 50, then the 51st document identifier in the inverted index can be used as the next document identifier to be read, that is, the second document identifier is D1.

[0073] Specifically, when reading the second document identifier based on the storage address of the second document identifier, the method of loading all data in the physical storage unit where the second document identifier is located into the second storage area can refer to the relevant description above of loading all data in the physical storage unit where the first document identifier is located into the second storage area based on the first document identifier, and will not be repeated here.

[0074] It is understandable that when performing document jump addressing based on document jump step size, the storage address of the second document identifier and the storage address of the previously read document identifier (such as the first document identifier) ​​belong to different physical storage units (such as physical pages), which can avoid repeated data retrieval and improve the efficiency of data reading.

[0075] Understandably, when loading all the data in the physical storage unit where the second document identifier is located into the second storage area, it is possible to check whether the entire inverted index of the target term has been loaded into the second storage area. If it is detected that the entire inverted index of the target term has been loaded into the second storage area, then the step of loading the inverted index of the target term into the second storage area is completed, which means that the inverted index of the target term is preheated.

[0076] Conversely, if it is detected that the entire inverted index of the target term has not been loaded into the second storage area, the second document identifier is updated based on the document jump step. Then, based on the storage address of the updated second document identifier, all data in the physical storage unit where the updated second document identifier is located is loaded into the second storage area, until all data in the physical storage unit where the second document identifier is located is loaded into the second storage area. Updating the second document identifier based on the document jump step can be described above in terms of the first document identifier and the document jump step. That is, the document identifier in the inverted index that is an interval of document jump step after the second document identifier can be used as the next document identifier to be read, and the determined next document identifier to be read is used to update the second document identifier; in other words, the next document identifier to be read is used as the updated document identifier.

[0077] For example, the inverted index of term C1 is [D1-D2-D3....D300], and the first document identifier is the first document identifier in the inverted index, i.e., the first document identifier is D1. If the document jump step size is 50, then the 51st document identifier in the inverted index can be used as the next document identifier to be read, i.e., the second document identifier is D1. Furthermore, after loading all the data in the physical storage unit Y1 where D1 is located into the second storage area, if the inverted index of the target term is not fully loaded into the second storage area, then based on document jump... Step size 50 determines that the 101st document identifier will be used as the next document identifier to be read, that is, the second document identifier will be updated to D101, and then all the data in the physical storage unit Y2 where D101 is located will be loaded into the second storage area; if the inverted index of the target term is not fully loaded into the second storage area, the 151st document identifier can be used as the next document identifier to be read based on the document jump step size 50, that is, the second document identifier will be updated to D151, and so on, until the entire inverted index is loaded into the second storage area.

[0078] Optionally, detecting whether the entire inverted index of the target term has been loaded into the second storage area can be done by determining whether the physical storage unit corresponding to the storage address of the last document identifier in the inverted index is the same as the physical storage unit where the second document identifier is located. If the physical storage unit corresponding to the storage address of the last document identifier in the inverted index is the same as the physical storage unit where the second document identifier is located, it means that the entire inverted index of the target term has been loaded into the second storage area. Conversely, if the physical storage unit corresponding to the storage address of the last document identifier in the inverted index is different from the physical storage unit where the second document identifier is located, it means that the entire inverted index of the target term has not been loaded into the second storage area.

[0079] Understandably, each term in the target term list can be used as a target term. Then, based on the above method, the inverted index of each term in the target term list can be loaded into the second storage area, which is to complete the preheating of the data retrieval node.

[0080] For example, see Figure 4 , Figure 4 This is a schematic diagram illustrating the effect of document skipping addressing provided in an embodiment of this application. For example... Figure 4 As shown, an inverted index of the target term can be stored in the first storage area (such as a disk). The document identifiers in the inverted index of the target term can include D0-D1000. Furthermore, the physical storage units (taking physical pages as an example) storing the inverted index in the first storage area can include physical page 1, physical page 2, ..., physical page n. The document identifiers stored in each physical page can be as follows: Figure 5 As shown in the diagram, the first document identifier in the inverted index of the target term is D0. Therefore, when reading the inverted index of the target term, the initial addressing can be to find the storage address of document identifier D0, which is stored in physical page 1. When document identifier D0 is read, physical page 1 can be loaded into the second storage area (that is, all data in physical page 1 is loaded into the second storage area). Further, the document jump step size can be obtained; here, we take a document jump step size Y=128 as an example. Then, document jump addressing (also called jump addressing) can be performed based on this document jump step size. That is, the document identifier for the next addressing (i.e., the second document identifier) ​​can be determined based on the document jump step size. Here, document identifier D128 can be read from physical page 2 (the number of document identifiers between document identifier D128 and document identifier D0 is equal to 128), thus loading physical page 2 into the second storage area. Furthermore, the document identifier for the next addressing (i.e., the second document identifier) ​​can be determined based on the document jump step size. Here, document identifier D256 can be read from physical page 3 (the number of document identifiers between document identifier D256 and document identifier D128 is equal to 128), and physical page 3 can be loaded into the second storage area. This process can be repeated until all the data of physical page n is loaded into the second storage area, thereby reading all the document identifiers in the inverted chain of the target term, which means loading the inverted chain of the target term into the second storage area (such as memory).

[0081] This section explains the document skipping addressing process using a flowchart. Please refer to [link / reference]. Figure 5 , Figure 5 This is a schematic diagram of a document skipping addressing process provided in an embodiment of this application. For example... Figure 6As shown, obtain the storage address of the first document identifier in the inverted list of the target term (step S501a), that is, obtain the storage address of the first document identifier. Further, load all the data in the physical storage unit where the first document identifier is located into the second storage area (step S502a), and then determine the next second document identifier to be read based on the document skip step and the first document identifier (step S503a). The specific method for determining the second document identifier can be referred to the above relevant description and will not be elaborated here. Furthermore, load all the data in the physical storage unit where the second document identifier is located into the second storage area (step S504a). Further, detect whether all the inverted list has been loaded into the second storage area (step S505a). If all the inverted list has been loaded into the second storage area, it is determined that the loading of the inverted list is completed (step S506a). On the contrary, if all the inverted list has not been loaded into the second storage area, update the second document identifier based on the document skip step and the previously read document identifier (step S507a). The way to update the second document identifier can be referred to the above relevant description and will not be elaborated here. Then load all the data in the physical storage unit where the second document identifier is located into the second storage area (step S504a) until all the inverted list has been loaded into the second storage area.

[0082] Based on this, in the scenario of preheating the inverted list of the search engine, a constraint mapping relationship between the logical document skip step Y, the physical page size P of the operating system, and the average number of bytes per DocID D of the inverted list (Y×D<P) can be established. Trigger a page fault through sparse addressing to achieve full physical page coverage with the least number of logical reads. Compared with full sequential reads, the disk (i.e., the first storage area) addressing times are reduced by 87.5%, and the physical layer page cache coverage rate remains 100% (without omission); the preheating speed is 2.1 to 16 times faster than full traffic playback, eliminating 49% of the memory cache noise, thus greatly improving the efficiency of data preheating.

[0083] In a possible implementation manner, the embodiment of the present application may further include the following steps: after each term in the target term list is used as the target term, load the inverted list of each term in the target term list into the second storage area to provide the target retrieval service based on the inverted list of each term in the target term list stored in the second storage area.

[0084] The target retrieval service can be a service used for retrieval. For example, this target retrieval service can be a service in a content platform that searches for published content, users, products, web pages, and other data based on the search text. It's understandable that the second storage area, used for temporary data storage in the data retrieval node, is faster than the first storage area when reading data. Therefore, after loading the terms from the target term list into the second storage area, when providing the target retrieval service, the inverted index of the terms in the search text corresponding to the query request can be read from the second storage area to more quickly determine the retrieval results.

[0085] In one possible implementation, the present application embodiment may further include the following steps: 1. When at least one term parsed from the query request is obtained, the inverted chain of each term in the query request is searched in the second storage area; 2. If the inverted chain of each term in the query request is found in the second storage area, document recall is performed based on the inverted chain of each term in the query request, so as to determine the retrieval result of the query request based on the recalled documents.

[0086] The query request can be a search request initiated by a user, and the query request can carry search information, which can be text data or voice data.

[0087] At least one term parsed from the query request can be obtained by the data access node. Upon receiving the query request, the data access node can determine the retrieval information from it, then determine the retrieval text based on the retrieval information, and perform word segmentation on the retrieval text to obtain at least one term. The retrieval text can be the text used for retrieval; it can be determined directly from the text data in the retrieval information, or it can be obtained by converting the speech data in the retrieval information into text; no limitation is made here.

[0088] Word segmentation can be a process of splitting a piece of text into multiple terms, that is, dividing a continuous text sequence into independent lexical units (i.e., terms) with clear semantics. For example, if the search query text is "fitness exercise", after word segmentation, the terms "fitness" and "exercise" can be obtained. Optionally, in this application, word segmentation can be performed using statistical methods such as HMM (Hidden Markov Model) and CRF (Conditional Random Field), or deep learning-based methods such as a combination of LSTM and CRF, or other methods, which are not limited here.

[0089] Furthermore, the data retrieval node can search for the inverted index of each term in the query request in the second storage area. If the inverted index of each term in the query request is found in the second storage area, document retrieval is performed based on the inverted index of each term in the query request, so as to determine the retrieval results of the query request based on the retrieved documents.

[0090] The search results can be the results obtained after searching based on the query request. Determining the search results based on the recalled documents can be done either by directly determining the search results based on the recalled documents, or by summarizing the content of the recalled documents using a large model after identifying the recalled documents to generate summary content specific to the query request, thereby determining the search results based on the recalled documents and the summary content.

[0091] Document retrieval based on the inverted list of each term in the query request may include: determining at least one candidate document identifier based on the document identifier in the inverted list of at least one term and the retrieval logic that matches the target retrieval service; and finally ranking the relevance between the document corresponding to the at least one candidate document identifier and the retrieval text using a relevance calculation algorithm (such as TF-IDF, BM25, etc.) to determine the recalled documents based on the ranking results.

[0092] The candidate document identifier can be the document identifier of a candidate retrievable document. The retrieval logic can be the logic used to retrieve documents, and this retrieval logic is matched to the target retrieval service. For example, if the retrieval logic indicates that the retrieved document must simultaneously include F terms (F is a positive integer) from at least one term, then when performing logical operations on the document identifiers in the inverted list of at least one term, the frequency of occurrence in multiple inverted lists can be counted, and document identifiers with a frequency of F can be used as candidate document identifiers; or, if the retrieval logic indicates that the retrieved document must simultaneously include F terms (F is a positive integer) from at least one term, then when performing logical operations on the document identifiers in the inverted list of at least one term, the frequency of occurrence in multiple inverted lists can be counted, and document identifiers with a frequency of F can be used as candidate document identifiers.

[0093] The relevance between a document and the search text indicates the degree of relevance between them. A higher relevance means the document is more relevant to the search text, and therefore more likely to be included as a recall document. A recall document refers to the document retrieved through document recall.

[0094] Optionally, the recall documents can be determined based on the sorting results, which can be by identifying the documents ranked F before sorting as the recall documents, where F is a positive integer.

[0095] Optionally, if the term in the query request is not found in the second storage area, it means that the inverted index of the term has not been loaded into the second storage area (possibly due to incomplete warm-up or the term not being in the target term list). In this case, the inverted index of the term can be read from the first storage area to enable document retrieval based on the inverted index of each term in the query request, and to determine the retrieval results of the query request based on the retrieved documents.

[0096] Using the embodiments of this application, a list of target terms can be obtained from a data storage object associated with a data access node, and the inverted index of each term in the target term list is stored in a first storage area. This target term list consists of at least one term determined by the data access node that satisfies a popularity condition, and this at least one term includes the target term. Further, the inverted index of the target term is searched in the second storage area of ​​the data retrieval node. If the inverted index of the target term is not found, a document jump step size satisfying a jump constraint condition is obtained. This document jump step size refers to the number of document identifiers skipped each time a document identifier in the inverted index of the target term is read from the first storage area. Then, based on the storage address of the document identifier in the inverted index of the target term and the document jump step size, document jump addressing is performed in multiple physical storage units included in the first storage area. Based on the addressed physical storage unit, the inverted index of the target term is loaded into the second storage area to provide data retrieval services based on the inverted index in the second storage area. Based on this, a list of target terms that meet the popularity criteria can be obtained through data access nodes, and the inverted chain of terms in the target term list can be moved from the first storage area to the second storage area through jump addressing, which helps to improve the efficiency of data preheating.

[0097] Based on the above description, this application proposes a data processing method. Please refer to [link to relevant documentation]. Figure 6 , Figure 7 This is a flowchart illustrating a data processing method provided in an embodiment of this application. The data processing method is applied to a data processing system, which includes a data access node and a data retrieval node. The method is executed by the data access node. The data processing method may include the following steps S201-S205.

[0098] S201. Create multiple threads for term counting; each thread maintains a buffer area; the multiple threads include the target thread.

[0099] Among them, term counting can refer to the statistical counting of terms obtained by segmenting the search text in each query request received by the data processing system.

[0100] For details regarding threads, please refer to the relevant descriptions above; they will not be repeated here. Creating multiple threads for term counting can help improve the efficiency of term counting.

[0101] Understandably, each thread in a multi-threaded system maintains its own buffer area, also known as a thread-private write buffer. This buffer area can be a temporary memory space within a thread's local storage area, balancing the speed difference between high-speed CPU processing and low-speed I / O operations. Specifically, this buffer area can temporarily store data to be written by the thread, and then submit it in batches when the buffer is full or certain conditions are met. This reduces the number of direct write operations, lowers system call overhead, and prevents threads from being blocked while waiting for slow write operations, thus improving overall operating efficiency.

[0102] The target thread can be any thread among multiple threads. In other words, each of the multiple target threads can be used as the target thread, so that the same operation can be performed on each thread (but the operation performed on each thread is independent, only the processing logic is the same, it does not mean that the same processing steps are performed synchronously on each thread) to complete the counting of terms (i.e. search terms) in the query request received by the data processing system.

[0103] S202. When the target thread obtains at least one term parsed from the target query request, it writes the count value of each term in the target query request into the target buffer area maintained by the target thread.

[0104] The target query request can be a query request received by the data access node that performs term counting through the target thread. It can be understood that any query request corresponding to a term obtained by the target thread can be called a target query request.

[0105] It is understandable that when a data access node receives a target query request, it can perform word segmentation on the search text in the target query request to obtain at least one term in the target query request. The method for determining at least one term in the target query request can be found in the relevant description above, and will not be repeated here.

[0106] The count value for each term in the target query request can refer to the number of each term in the search text of the target query request. For example, if the search text in the target query request is "travel outfit", then the terms obtained by segmenting the search text can include "travel" and "outfit". In this target query request, the count value of the term "travel" is 1, and the count value of the term "outfit" is 1.

[0107] Optionally, if terms appear repeatedly in the target query request, the count value of each term in the target query request can be obtained by counting the number of identical terms in the target query request, or the count value of each term in the target query request can be counted separately. There is no limitation here. For example, if the search text in the target query request is "recommendation of a three-day travel guide for region A", then the terms obtained from word segmentation of this search text can include "region A", "of", "three", "day", "travel", "guide", "of", and "recommendation". If the number of identical terms in the target query request is counted, then the count value of each term in the target query request can include: "region A" count value is 1, "of" count value is 2, "three" count value is 1, "day" count value is 1, "travel" count value is 1, "guide" count value is 1, and "recommendation" count value is 1. Alternatively, the count value of each term in the target query request can be counted separately, then the count value of each term in the target query request can include: "region A" count value is 1, "of" count value is 1, "three" count value is 1, "day" count value is 1, "travel" count value is 1, "guide" count value is 1, "of", and "recommendation" count value is 1.

[0108] The target buffer area can be a buffer area maintained by the target thread. Understandably, the count value of each term in the target query request can be written into the target buffer area maintained by the target thread. For example, if the target buffer area originally records data as [food+1, travel+1, food+1, skincare+1, food+1, travel+1...], and in the obtained target query request, the count value of the term "travel" is 1, and the count value of the term "outfit" is 1, then after writing the count values ​​of the terms in the target query request into the target buffer area, the data in the target buffer area can be: [travel+1, outfit+1, food+1, travel+1, food+1, skincare+1, food+1, travel+1...].

[0109] As can be understood from the above description, each thread maintains a corresponding buffer area, and each buffer area can be used to count the terms in the retrieved query request.

[0110] Therefore, when a query request arrives, the term count value of the query request can be written to the private buffer area of ​​the corresponding thread. Based on this operation, each thread can be used to record the term count value of the obtained query request. The buffer area of ​​each thread can be as follows: Thread 1 buffer area: [Food +1, Travel +1, Food +1, Skincare +1, ...]; Thread 2 buffer area: [Food +1, Outfit +1, Food +1, ...]; Thread N buffer area: [Outfit +1, Sports +1, Swimming +1, ...].

[0111] Understandably, in this application, the term count update operation is only appended to the thread's private buffer area. This stage involves no lock operations and no inter-thread memory synchronization (thus eliminating the need for CPU cache coherency protocol overhead). Regardless of the number of concurrent threads or the QPS, the overhead of each write operation is consistently O(1) local memory append. Based on this, write requests that would otherwise compete for the global lock are completely moved to the thread's private space, eliminating lock contention at the source and thus resolving the write storm that might be caused by query requests.

[0112] S203. When the aggregation condition of the target buffer area is met, the count value of each term in the target buffer area is aggregated to obtain the aggregated count value of at least one candidate term.

[0113] The aggregation condition can refer to the condition for aggregating the count value of each term in the target buffer region. It is understood that this aggregation condition can be a constraint on the number of terms in the target buffer region, and / or a constraint on time, which is not limited here.

[0114] In one possible implementation, the aggregation conditions include one or more of the following: the number of counts in the target buffer region reaches the capacity threshold of the target buffer region, and the current time reaches a preset aggregation time.

[0115] In this context, a count item refers to the data used for term counting in the target buffer region. A term and its corresponding count value are called a count item. For example, in the thread 1 buffer region: [Food +1, Travel +1, Food +1, Skincare +1, ...], Food +1 is a count item, and Travel +1 is a count item. It's understandable that the target buffer region can only hold a limited number of count items; that is, the target buffer region has a capacity threshold. This capacity threshold refers to the maximum number of count items the target buffer region can hold. Therefore, when the number of count items in the target buffer region reaches the capacity threshold, it indicates that the aggregation condition of the target buffer region has been met. Consequently, aggregate counting can be performed on the terms in the target buffer region (i.e., aggregate counting can be performed on the count value of each term in the target buffer region).

[0116] Here, "current time" can refer to the latest, standard time obtained by the system in real time. "Preset aggregation time" refers to the time at which the count values ​​of terms in the target buffer area are aggregated and counted. This can be understood as the time interval between the last aggregation and count of terms in the target buffer area, allowing for periodic aggregation and counting at regular intervals. For example, if the preset aggregation time is 1 second, then if the last aggregation and count occurred at time T1, the preset aggregation time would be T1+1 seconds (denoted as T2), allowing for aggregation and counting of terms in the target buffer area when the current time reaches T2.

[0117] The aggregation counting process involves deduplicating terms in the target buffer region and merging the counts of identical terms. Specifically, during aggregation counting, at least one candidate term is obtained by deduplicating terms in the target buffer region, and the counts corresponding to these candidate terms are aggregated to obtain the aggregate count value for the candidate term. The candidate term can be any term obtained by deduplicating terms in the target buffer region, and this candidate term can be used as a candidate for the target term list. The aggregate count value is the numerical value obtained by accumulating the counts of the candidate terms in the target buffer region.

[0118] In one possible implementation, the count value of each term in the target buffer region is aggregated to obtain the aggregated count value of at least one candidate term, including: merging identical terms in the target buffer region to obtain at least one candidate term, and adding the count values ​​corresponding to each candidate term to obtain the aggregated count value of each candidate term.

[0119] Merging identical terms can refer to combining multiple identical terms into a single term (i.e., a candidate term). It's understandable that if a term in the target buffer region does not have an identical term, then that term can be directly used as a candidate term.

[0120] If a candidate term is obtained by merging multiple identical terms, then the count values ​​corresponding to the multiple identical terms can be added together to obtain the aggregate count value of the candidate term; if a candidate term is not obtained by merging multiple identical terms, but is directly obtained from a term, then the count value of that term can be used as the aggregate count value of the candidate term.

[0121] For example, before aggregation counting, the target buffer area contains 256 count items, such as [food +1, travel +1, food +1, food +1, skincare +1, food +1, travel +1,...]. After aggregation counting, the same items in the target buffer area are merged to obtain at least one candidate item, and the count values ​​of the same items are added together to obtain the aggregation count value. Generally speaking, only a few dozen items remain after aggregation, such as the candidate items and count values ​​after aggregation counting: [food +80, travel +35, skincare +20, fashion +15,...].

[0122] S204. Write the aggregated count value of at least one candidate term into the global counting structure, obtain at least one term that meets the popularity condition based on the global counting structure, and determine the target term list based on at least one term that meets the popularity condition.

[0123] The global counting structure can be used to count terms collected by multiple threads. This global counting structure can include at least one counting slice, each of which can be used to count terms. Each counting slice can be based on a traditional array, hash table, or bitmap, or it can be based on a probability decay counting structure; no specific limitation is made here.

[0124] The popularity condition can be used to select candidate terms as terms in the target term list, that is, to designate candidate terms as hot words. This popularity condition can refer to the candidate term's global count value being among the top K, and / or the candidate term's global count value being greater than or equal to a preset value. The global count value can refer to the numerical value obtained by counting candidate terms using a global counting structure.

[0125] Understandably, in this application, the terms for each query request within a thread can be aggregated and counted internally (this process does not require any global locks). Then, the aggregated count results (i.e., the aggregated count value of at least one candidate term) are written to the global counting structure. In this way, the amount of data after aggregation and counting is typically compressed to less than 1 / 10 of the original (because hot terms appear repeatedly, resulting in a higher merging ratio). For the results of multiple terms from multiple query requests, only one lock operation is needed on the global lock of the global counting structure. After writing the aggregated count results to the global counting structure, the lock can be released. Therefore, compared to directly recording the term count value for each query request separately through the global counting structure, a single global lock operation replaces multiple global lock operations (i.e., the capacity of each thread's buffer area, such as 256 times), avoiding a "write storm" during term counting (i.e., the abnormal phenomenon of a massive influx of concurrent write requests far exceeding the normal load in a short period, directly overwhelming the storage and query links). Furthermore, the data written to the global counting structure is already aggregated and counted, which further reduces the update pressure on the global counting structure.

[0126] For example, the following are the quantization effects caused by different methods using a global counting structure:

[0127] In one possible implementation, the aggregated count value of at least one candidate word is written into a global counting structure, and at least one word that meets the popularity condition is obtained based on the global counting structure. This may include the following steps: 1. Obtaining the aggregated count value of at least one candidate word; the at least one candidate word includes a target candidate word; 2. Based on the hash value corresponding to the target candidate word, determining a target counting segment from at least one counting segment in the global counting structure for counting the target candidate word; the target counting segment includes the global count value corresponding to at least one candidate word; 3. Updating the global count value of the candidate word recorded in the target counting segment with the aggregated count value of the target candidate word; 4. When the hot word statistics condition of the global counting structure is met, determining at least one word that meets the popularity condition based on the global count value of the candidate word recorded in at least one counting segment.

[0128] The aggregate count value of at least one candidate term may refer to the aggregate count value of candidate terms stored in the buffer area of ​​the target thread, so as to write the aggregate count value of candidate terms in the target thread into the global count structure.

[0129] The target candidate word can be any one of at least one candidate word. The hash value of the target candidate word can be a value obtained through hash calculation.

[0130] The target counting segment can be a segment used to count target candidate words. The target counting segment includes a global count value corresponding to at least one candidate word.

[0131] The target counting slice can be determined based on the hash value of the target candidate word. Specifically, the hash value of the target candidate word can be divided modulo the number of counting slices to determine the counting slice represented by the remainder as the target counting slice.

[0132] For example, the method for determining the target count slice can be determined by the following formula (2): ShardIndex=Hash(Key)%N formula (2) Here, ShardIndex can be the identifier of the target counting shard. Key represents the target candidate word, and Hash(Key) is the hash value of the target candidate word. N represents the number of counting shards in the global counting structure.

[0133] The global count value of candidate terms recorded in the target counting segment is updated in a way that is adapted to the counting method of the target counting segment. The specific method can be found in the following related descriptions, which will not be elaborated here.

[0134] Among them, the hot word statistics condition can refer to the condition for determining at least one word item that meets the heat condition based on the data recorded in the global counting structure.

[0135] The hot word statistics condition can be that the current account reaches a preset statistical time, which can be the time when keywords meeting the popularity criteria are reported to the offline calculation task. This preset statistical time can also be the time interval between the last report and the current report time; for example, a report can be performed at regular intervals (e.g., 2 hours). For instance, if the preset aggregation time is 2 hours, then if the last aggregation count was performed at time t1, the preset aggregation time would be t1 + 2 hours (e.g., t2). This allows the keywords meeting the popularity criteria in the current global counting structure to be determined when the current time reaches t2.

[0136] Optionally, the target term list can be determined based on at least one candidate term that meets the popularity criteria. This can be done by directly using at least one candidate term that meets the popularity criteria as the target term list, or by merging at least one candidate term that meets the popularity criteria determined by the current data access node with at least one candidate term that meets the popularity criteria determined by other data access nodes to obtain the target term list.

[0137] Optionally, the current data access node can determine the current Top-K hot word list (i.e., a list consisting of at least one word that meets the popularity criteria) through a separate background thread at a preset statistical time (e.g., every 2 hours), serialize it into structured data, and then the event manager asynchronously reports the structured data of the Top-K hot word list to the offline data stream through the event log channel (EventLogger), without writing to the local disk and without affecting the disk read / write of the current node. The offline data stream is a type of data processing link in the big data field, whose core is to complete the migration, cleaning, and calculation of data in batches at fixed periods, which differs from the continuous low-latency processing of real-time data streams.

[0138] Furthermore, the data storage object can process the event log data streams (i.e., offline data streams) of each data access node through offline computing tasks. It can then merge and aggregate the Top-K hot word lists (also known as node hot word lists) sent by multiple data access nodes according to cluster identifier and date dimension, obtaining a snapshot file of the target term list. This snapshot file can be archived and stored on a remote object by date. After all the fragmented data of the snapshot file has been written, a version completion marker file (such as a file with the .done extension) is written to ensure that the data is complete and consistent when the data retrieval node reads the snapshot file.

[0139] Optionally, the structured data of the Top-K hot term list (node ​​hot term list) may include: cluster identifier, index field name, statistics date, and Top-K term list. The cluster identifier can be used to uniquely identify the retrieval cluster, distinguishing different clusters. For example, different clusters could be clusters used for product searches or clusters used for content searches; this is not limited here. The index field name can refer to the field that defines the Top-K hot term list. For example, this index field name indicates that the Top-K hot term list is based on user-inputted search terms (such as user_input_query). The statistics date can indicate the date the Top-K hot term list was determined. The Top-K term list can refer to at least one term from the Top-K terms.

[0140] Optionally, if the counting segments in the global counting structure are based on traditional arrays, hash tables, or bitmaps for counting, then updating the global count value of candidate terms in the target counting segment can be done by finding the global count value of the target candidate term in the target counting segment and adding the global count value of the target candidate term to the aggregate count value to obtain the updated global count value of the target candidate term. Furthermore, when the hot word statistics condition is met, based on the global count values ​​of candidate terms recorded in each counting segment of the global counting structure, at least one candidate term that meets the popularity condition is selected. For example, the candidate terms whose global count values ​​rank in the top K are considered as at least one candidate term that meets the popularity condition, which can also be called top-k hot words.

[0141] Optionally, if the counting segments in the global counting structure are based on a probability decay structure, then during the counting process through the target counting segment, candidate words with low global count values ​​will be discarded. This ensures that all candidate words recorded in the global counting structure meet the popularity criteria. Consequently, when the hot word statistics criteria are met, the candidate words recorded in each counting segment of the global counting structure are directly used as at least one candidate word that meets the popularity criteria, which can also be called top-k hot words.

[0142] In one possible implementation, the aggregated count value of the target candidate term is written into the target counting segment, and the global count value of the target candidate term is obtained by statistical analysis of the target counting segment. This may include the following steps: 1. If no term identifier corresponding to the target candidate term is found in the target counting segment, the global count value of each historical candidate term stored in the target counting segment is obtained, and the decay probability of each historical candidate term is calculated based on the global count value of each historical candidate term; 2. The global count value of each historical candidate term is updated based on the decay probability of each historical candidate term; 3. If there is a target historical candidate term in the target counting segment whose updated global count value is less than or equal to a preset value, the target historical candidate term is replaced with the target candidate term, and the aggregated count value of the target candidate term is used as the global count value of the target candidate term.

[0143] The term identifier can be an identifier used to uniquely identify a term in the target counting segment. It can be the hash value of the candidate term or the fingerprint of the candidate term (such as a 64-bit hash value).

[0144] Optionally, the target counting segment can be counted using a fingerprint bucket array. In the fingerprint bucket array, each cell can be used to store the term identifier (i.e., fingerprint) and global count value of the candidate term. Optionally, the representation of the term identifier and the global count value can be preset. For example, the term identifier can be a 64-bit hash value, and the global count value can be a 32-bit integer.

[0145] The decay probability can be the probability used to decay the corresponding global count value. This decay probability can be calculated based on the global count values ​​of historical candidate terms. The decay probability is inversely proportional to the global count value; that is, the larger the global count value, the lower the decay probability, and vice versa.

[0146] For example, this attenuation probability can be calculated using formula (3): Formula (3) Where P is the decay probability of historical candidate terms. This is the global count value of historical candidate terms. This is a preset constant, for example, b=1.08.

[0147] Specifically, for each historical candidate term, the global count value of the historical candidate term can be updated based on the decay probability of the historical candidate term, and the probability of updating is the decay probability.

[0148] Updating the global count of historical candidate terms can be achieved by reducing the global count of historical candidate terms by a preset decay value. This preset decay value can be the amount reduced each time the global count is decayed, for example, the preset decay value is 1.

[0149] The target historical candidate term can be a historical candidate term whose updated global count value is less than or equal to a preset value. This preset value can be a pre-defined numerical value used to determine which historical candidate term will be replaced; for example, the preset value could be 0. It is understood that if a target historical candidate term exists in the target count segment with an updated global count value equal to the preset value, it indicates that the probability of the target historical candidate term's count value being in the top K is low. In this case, the target historical candidate term is replaced with the target candidate term, and the aggregated count value of the target candidate term is used as the global count value of the target candidate term.

[0150] It is understandable that if there is no target historical candidate word in the target counting segment whose updated global count value is less than or equal to the preset value, the target candidate word can be discarded, that is, the target candidate word is not written into the target counting segment.

[0151] In one possible implementation, the aggregated count value of the target candidate term is written into the target count segment, and the global count value of the target candidate term is obtained by statistical analysis of the target count segment. This may include the following steps: if the term identifier corresponding to the target candidate term is found in the target count segment, the global count value of the target candidate term is updated based on the aggregated count value of the target candidate term.

[0152] If the target candidate term identifier is found in the target counting segment, it means that the global count value of the target candidate term has been recorded in the target counting segment. Then, the global count value of the target candidate term can be updated directly.

[0153] Updating the global count of the target candidate term based on the aggregate count of the target candidate term can be achieved by adding the aggregate count to the global count of the target candidate term to obtain the updated global count.

[0154] For example, see Figure 7 , Figure 7 This is a schematic diagram illustrating the effect of hot word statistics provided in an embodiment of this application. For example... Figure 7 As shown, after receiving a query request, at least one term can be parsed from the query request. Further, step 1 can be executed: internal thread storage, for example, through thread 1 (such as...). Figure 7 As shown in 701a), Thread 2 (as shown in 701a) Figure 7 As shown in 702a), ..., thread N (as shown in ... Figure 7 As shown in 703a, term counting (i.e., recording the term count value in each query request) is performed. Each thread can maintain a corresponding buffer area, and each thread operates independently. Then, step 2: in-thread aggregation counting can be executed. Specifically, for each thread, it can be detected in real time whether the aggregation condition has been met (e.g., ...). Figure 7 As shown in 704a), it can detect whether the number of count items has reached the capacity threshold (e.g. Figure 7 (as shown in 705a), or whether the current time has reached the preset aggregation time (e.g.) Figure 7 (As shown in 706a). Furthermore, when the number of count items reaches the capacity threshold, or when the current time reaches the preset aggregation time, the count can be aggregated within the thread (e.g., ...). Figure 7 As shown in 707a), the aggregated count results are obtained (e.g. Figure 7As shown in 708a), the aggregated count result can include at least one candidate term in the thread and the aggregated count value corresponding to each candidate term. The method of performing aggregated counting within the thread can be found in the relevant description above, and will not be repeated here. Further, step 3: global counting can be performed. Specifically, for each candidate term in the aggregated count result, hash routing (such as...) can be used. Figure 7 (As shown in 709a) from the global counting structure (such as...) Figure 7 As shown in 710a, a counting slice is determined from multiple counting slices in the structure to record the global count value of candidate terms. Among these, the global counting structure (such as...) Figure 7 The multiple counting slices in (as shown in 710a) may include counting slice 1 (such as...) Figure 8 As shown in 711a), counting slice 2 (as shown in 711a). Figure 8 As shown in 712a), ..., counting segments M (as shown in 712a) Figure 8 As shown in 713a), each counting segment can be counted by fingerprint bucket arrays (such as fingerprint bucket array 1 in counting segment 1, fingerprint bucket array 2 in counting segment 2, or fingerprint bucket array M in counting segment M, etc.), and top-K hot words can be recorded by decaying counting mechanism. The specific decaying counting mechanism can be found in the relevant description above, and will not be elaborated here.

[0155] This section illustrates the counting process based on a decay mechanism using diagrams. Please refer to [link / reference]. Figure 9 , Figure 9 This is a flowchart illustrating a counting mechanism for counting segments provided in an embodiment of this application. For example... Figure 9As shown, the input can first be: Term C1: Count value Z1 (step 801a), where term C1 can be any candidate term from at least one candidate term in the above aggregated counting results. Further, it is checked whether a matching term identifier exists in the fingerprint bucket array (step 802a). If a matching term identifier exists in the fingerprint bucket array, the count value of term C1 is added to the count value of Z1 (step 803a). If no matching term identifier exists in the fingerprint bucket array, the count values ​​of historical candidate terms in the counting segment can be obtained (step 804a). Then, the decay probability of each historical candidate term is calculated (step 805a), and then, based on the decay probability, it is determined whether decay occurs (step 806a). For any historical candidate term, if it is determined based on the decay probability that no decay occurs, then the count value of the historical candidate term remains unchanged (step 807a). Conversely, if attenuation is determined to have occurred based on the attenuation probability, the count value of the historical candidate term is reduced by a preset attenuation value (step 808a). Then, it is checked whether the count value after reducing the preset attenuation value is less than or equal to the preset value (step 809a). If the count value after reducing the preset attenuation value is less than or equal to the preset value (e.g., less than or equal to 0), the historical candidate term is replaced with term C1, and the count value of term C1 is count value Z1 (step 810a). Conversely, if the count value after reducing the preset attenuation value is greater than the preset value (e.g., greater than 0), term C1 is discarded (step 811a).

[0156] S205. Store the target term list in a data storage object related to the data access node so that the data retrieval node can obtain the target term list from the data storage object.

[0157] For details regarding data storage objects, please refer to the relevant descriptions above; they will not be repeated here.

[0158] Furthermore, the data retrieval node can obtain the target term list from the data storage object, store the inverted index of each term in the target term list into the first storage area of ​​the data retrieval node, and complete the warm-up of the data retrieval node. The method for warming up the data retrieval node can be found in the relevant description above, and will not be repeated here.

[0159] It is understandable that for each service node in the aforementioned service cluster, the processing can be performed through decoupled modules. The flow between these modules is described here with reference to the diagram. Please refer to [link / reference]. Figure 9 , Figure 9 This is a schematic diagram illustrating the effect of a data processing system provided in an embodiment of this application. Figure 9 As shown, the data access node (i.e., node 101a) can receive query requests sent by the clients corresponding to each user (such as...).Figure 9 As shown in 901a), furthermore, it can be done by requesting the parsing module (such as...). Figure 9 As shown in 902a, the query request is parsed to obtain at least one term from the query request. Furthermore, a hot word statistics module (such as...) can be used to... Figure 9 As shown in 903a, the count of terms in a statistical query request can be calculated using multiple threads (such as...). Figure 9 As shown in 904a, threads 1, 2, N, etc., are used to record the count values ​​of terms in the query request. Each thread maintains a corresponding buffer area to record the count values ​​of terms in the query request. Then, the counts recorded in the buffer area can be aggregated, and the aggregation result can be written to a global counting structure (such as...). Figure 9 As shown in 905a), it can be accessed through a global counting structure (such as...). Figure 9 As shown in 905a), at least one term that meets the popularity criteria can be statistically obtained. Furthermore, this can be achieved through an event reporting manager (such as...). Figure 9 As shown in 906a, offline data streams are sent to the offline processing node (i.e., node 102a) so that the offline processing node can perform offline processing tasks (such as...). Figure 9 As shown in 907a, the target term list is determined, and a snapshot file of the target term list is determined, so as to store the snapshot file of the target term list into a data storage object (such as...). Figure 9 As shown in 908a). Furthermore, the data retrieval node (i.e., node 103a) can be controlled by the preheating control manager (e.g., ...). Figure 10 As shown in 909a, the preheating control manager retrieves a snapshot file of the target term list. Furthermore, upon triggering a page fault, the preheating control manager can determine the document jump step size and, based on this step size, identify the document to be read, thereby invoking the jump read executor (such as...). Figure 10 As shown in 910a, the inverted index is moved from the first storage area (e.g., Figure 11 As shown in 911a, the data is loaded into the second storage area (e.g., ...). Figure 11 As shown in 912a (the specific loading method can be found in the relevant description above). The method by which the preheating control manager determines the document identifier to be read can be found in the relevant description above regarding the determination of the first and second document identifiers, and will not be repeated here.

[0160] By employing the embodiments of this application, a three-stage write dimensionality reduction mechanism (i.e., a multi-level buffered aggregation counting mechanism) can be used to transform the high-frequency write operation that originally competed for the global lock with each query into an extremely low-frequency batch write operation. This mechanism, combined with a probability decay counting structure with fixed memory, solves the problem of "losslessly embedding streaming frequency statistics in high-concurrency search query chains." This reduces the number of global lock contention times by 99.96% (from 100,000 QPS to 39 times / second in a 100,000 QPS scenario); furthermore, the P99 time consumption of the main query chain is zero-perceptible; and the memory usage is strictly bounded, fundamentally eliminating the potential risk of memory overflow (OutOfMemoryError) that occurs with traditional hash tables during program execution. Based on this, a list of target terms that meet the popularity criteria can be obtained through data access nodes, and the inverted chain of terms in the target term list can be moved from the first storage area to the second storage area through jump addressing, which helps to improve the efficiency of data preheating.

[0161] Please see Figure 11 , Figure 11 This is a timing interaction diagram of a data processing method provided in an embodiment of this application. The data processing method involves a data access node, a data storage object, and a data retrieval node. The data processing method may include the following steps: S1001. Create multiple threads for term counting; S1002. When the target thread obtains at least one term parsed from the target query request, it writes the count value of each term in the target query request into the target buffer area maintained by the target thread. S1003. When the aggregation condition of the target buffer area is met, the count value of each term in the target buffer area is aggregated and counted to obtain the aggregated count value of at least one candidate term. S1004. Write the aggregated count value of at least one candidate term into the global counting structure, obtain at least one term that meets the popularity condition based on the global counting structure, and determine the target term list based on at least one term that meets the popularity condition. S1005. Store the target term list into a data storage object related to the data access node; S1006. Retrieve the target term list from the data storage object; S1007. Store the inverted chain of each term in the target term list in the first storage area; S1008. Search for the inverted chain of the target term in the second storage area of ​​the data retrieval node, and if the inverted chain of the target term is not found, obtain the document jump step size that satisfies the jump constraint. S1009. Based on the storage address of the document identifier in the inverted list of the target term and the document jump step size, perform document jump addressing in multiple physical storage units contained in the first storage area, and load the inverted list of the target term into the second storage area based on the addressed physical storage unit, so as to provide data retrieval services based on the inverted list in the second storage area.

[0162] Using the embodiments of this application, a list of target terms that meet the popularity criteria can be obtained through the data access node, and the inverted chain of terms in the target term list can be moved from the first storage area to the second storage area through a jump addressing method, thereby helping to improve the efficiency of data preheating.

[0163] Please see Figure 12 , Figure 12 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Figure 12 As shown, the data processing device 1 is applied to a data processing system, which includes a data access node and a data retrieval node. The device can be a computer program (including program code) running on the data retrieval node (such as node 103a mentioned above), for example, the data processing device 1 is an application software. It is understood that the data processing device 1 can be used to execute corresponding steps in the data processing method provided in the embodiments of this application. Figure 12 As shown, the data processing device 1 may include: a list acquisition unit 11, a step size acquisition unit 12, and a jump addressing unit 13; The list acquisition unit 11 is used to acquire a list of target terms from a data storage object related to the data access node, and store the inverted index of each term in the list of target terms in a first storage area; the list of target terms is at least one term that satisfies the popularity condition determined by the data access node; the at least one term includes the target term; The step size acquisition unit 12 is used to search for the inverted chain of the target term in the second storage area of ​​the data retrieval node, and when the inverted chain of the target term is not found, to acquire the document jump step size that satisfies the jump constraint condition; the document jump step size refers to the number of document identifiers skipped each time the document identifiers in the inverted chain of the target term are read from the first storage area. The skip addressing unit 13 is used to perform document skip addressing in multiple physical storage units included in the first storage area based on the storage address of the document identifier in the inverted list of the target term and the document skip step size, and load the inverted list of the target term into the second storage area based on the addressed physical storage unit, so as to provide data retrieval services based on the inverted list in the second storage area.

[0164] In one implementation, the step size acquisition unit 12 is specifically used for: Obtain the unit storage amount of a single physical storage unit, and determine the average identifier data amount based on the amount of stored data of document identifiers in the inverted chain; A reference value is calculated based on the unit storage capacity and the average identifier data volume; A jump constraint is constructed based on the reference value; the jump constraint is that the document jump step size is less than the reference value. The document jump step size that satisfies the jump constraint is determined from the values ​​that are smaller than the reference value.

[0165] In one implementation, the jump addressing unit 13 is specifically used for: Obtain the storage address of the first document identifier in the inverted chain of the target term in the first storage area; when reading the first document identifier based on the storage address of the first document identifier, load all data in the physical storage unit where the first document identifier is located into the second storage area; the first document identifier is the first document identifier in the inverted chain of the target term; The second document identifier is determined based on the document jump step size and the first document identifier, and is read in the inverted chain of the target term; the number of document identifiers that differ between the second document identifier and the first document identifier is the document jump step size; Obtain the storage address of the second document identifier in the first storage area. When reading the second document identifier based on the storage address of the second document identifier, load all data in the physical storage unit where the second document identifier is located into the second storage area. If the inverted index of the target term is not fully loaded into the second storage area, update the second document identifier based on the document jump step size to load the inverted index of the target term into the second storage area.

[0166] In one implementation, the jump addressing unit 13 is also used for: After treating each term in the target term list as a target term, the inverted index of each term in the target term list is loaded into the second storage area to provide target retrieval services based on the inverted index of each term in the target term list stored in the second storage area.

[0167] In one implementation, the data processing device 1 further includes: a data retrieval unit 14; Data retrieval unit 14 is specifically used for: If an inverted index for each term in the query request is found in the second storage area, document retrieval is performed based on the inverted index for each term in the query request, so as to determine the retrieval result of the query request based on the retrieved documents.

[0168] Please see Figure 13 , Figure 13 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Figure 13 As shown, the data processing device 2 is applied to a data processing system, which includes a data access node and a data retrieval node. The device can be a computer program (including program code) running on the data access node (such as node 101a mentioned above), for example, the data processing device 2 is an application software. It is understood that the data processing device 2 can be used to execute corresponding steps in the data processing method provided in the embodiments of this application. Figure 13 As shown, the data processing device 2 may include: a thread creation unit 21, a thread counting unit 22, an aggregation counting unit 23, a global counting unit 24, and a list storage unit 25; The thread creation unit 21 is used to create multiple threads for word counting; each of the multiple threads maintains a buffer area; the multiple threads include a target thread; The thread counting unit 22 is used to write the count value of each term in the target query request into the target buffer area maintained by the target thread when the target thread obtains at least one term parsed from the target query request; The aggregation counting unit 23 is used to aggregate the count value of each term in the target buffer area when the aggregation condition of the target buffer area is met, so as to obtain the aggregation count value of at least one candidate term. The global counting unit 24 is used to write the aggregated count value of at least one candidate term into the global counting structure, statistically obtain at least one term that meets the popularity condition based on the global counting structure, and determine the target term list based on at least one term that meets the popularity condition. The list storage unit 25 is used to store the target term list in a data storage object related to the data access node, so that the data retrieval node can obtain the target term list from the data storage object.

[0169] In one implementation, the aggregation conditions include one or more of the following: the number of counts in the target buffer region reaches the capacity threshold of the target buffer region, and the current time reaches a preset aggregation time.

[0170] In one implementation, the aggregation counting unit 23 is specifically used for: Merge identical terms in the target buffer region to obtain at least one candidate term, and sum the count values ​​corresponding to each candidate term to obtain the aggregate count value of each candidate term.

[0171] In one implementation, the global counting unit 24 is specifically used for: Obtain the aggregate count value of the at least one candidate term; the at least one candidate term includes the target candidate term; Based on the hash value corresponding to the target candidate term, a target counting segment for counting the target candidate term is determined from at least one counting segment in the global counting structure; the target counting segment includes the global count value corresponding to at least one candidate term; The aggregated count value of the target candidate term is used to update the global count value of the candidate terms recorded in the target count segment; When the hot word statistics condition of the global counting structure is met, at least one word that meets the heat condition is determined based on the global count value of the candidate words recorded in the at least one counting segment.

[0172] In one implementation, the global counting unit 24 is also used for: If the term identifier corresponding to the target candidate term is not found in the target counting segment, then the global count value of each historical candidate term stored in the target counting segment is obtained, and the decay probability of each historical candidate term is calculated based on the global count value of each historical candidate term. The global count value of each historical candidate word is updated based on the decay probability of each historical candidate word. If there is a target historical candidate word in the target counting segment whose updated global count value is less than or equal to a preset value, then the target historical candidate word is replaced with the target candidate word, and the aggregated count value of the target candidate word is used as the global count value of the target candidate word.

[0173] In one implementation, the global counting unit 24 is also used for: If the term identifier corresponding to the target candidate term is found in the target counting segment, the global count value of the target candidate term is updated based on the aggregate count value of the target candidate term.

[0174] Please see Figure 13 , ​ This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. For example, this computer device can be the terminal device or server described above. ​ As shown, the computer device 1000 may include a processor 1001, a network interface 1004, and a memory 1005. Furthermore, the computer device 1000 may also include a user interface 1003 and at least one communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as at least one disk storage device. Optionally, the memory 1005 may also be at least one storage device located remotely from the processor 1001. ​ As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

[0175] In such ​ In the computer device 1000 shown, the network interface 1004 provides network communication functionality; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to execute the data processing method described in any of the corresponding embodiments above, which will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0176] Furthermore, it should be noted that this application also provides a computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor, so that a computer device having a processor performs the methods described in the preceding embodiments. Therefore, these will not be repeated here. Additionally, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the embodiments of the computer-readable storage medium involved in this application, please refer to the description of the method embodiments of this application.

[0177] The aforementioned computer-readable storage medium can be the apparatus provided in any of the foregoing embodiments or the internal storage unit of the aforementioned computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device. Furthermore, the computer-readable storage medium may include both internal storage units and external storage devices of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0178] Furthermore, it should be noted that this application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The computer program is adapted to be read and executed by a processor, so that a computer device having a processor performs the method provided in any of the corresponding embodiments described above. Additionally, the beneficial effects of using the same method will not be repeated here. For technical details not disclosed in the computer program product or computer program embodiments involved in this application, please refer to the description of the method embodiments of this application.

[0179] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data processing method, characterized in that, The method is applied to a data processing system, which includes a data access node and a data retrieval node; the method is executed by the data retrieval node, and the method includes: A list of target terms is obtained from a data storage object associated with the data access node, and an inverted index of each term in the list of target terms is stored in a first storage area; the list of target terms is at least one term that satisfies the popularity condition determined by the data access node; the at least one term includes the target term. The inverted list of the target term is searched in the second storage area of ​​the data retrieval node, and if the inverted list of the target term is not found, the document jump step size that satisfies the jump constraint is obtained; the document jump step size refers to the number of document identifiers skipped each time the document identifiers in the inverted list of the target term are read from the first storage area. Based on the storage address of the document identifier in the inverted list of the target term and the document jump step size, document jump addressing is performed in multiple physical storage units contained in the first storage area. Based on the addressed physical storage unit, the inverted list of the target term is loaded into the second storage area to provide data retrieval services based on the inverted list in the second storage area.

2. The method according to claim 1, characterized in that, The step of obtaining the document jump step size that satisfies the jump constraint includes: Obtain the unit storage amount of a single physical storage unit, and determine the average identifier data amount based on the amount of stored data of document identifiers in the inverted chain; A reference value is calculated based on the unit storage capacity and the average identifier data volume; A jump constraint is constructed based on the reference value; the jump constraint is that the document jump step size is less than the reference value. The document jump step size that satisfies the jump constraint is determined from the values ​​that are smaller than the reference value.

3. The method according to claim 1, characterized in that, The step of performing document skipping addressing within multiple physical storage units in the first storage area based on the storage address of the document identifier in the inverted list of the target term and the document skipping step, and loading the inverted list of the target term into the second storage area based on the addressed physical storage unit, includes: Obtain the storage address of the first document identifier in the inverted chain of the target term in the first storage area; when reading the first document identifier based on the storage address of the first document identifier, load all data in the physical storage unit where the first document identifier is located into the second storage area; the first document identifier is the first document identifier in the inverted chain of the target term; The second document identifier is determined based on the document jump step size and the first document identifier, and is read in the inverted chain of the target term; the number of document identifiers that differ between the second document identifier and the first document identifier is the document jump step size; Obtain the storage address of the second document identifier in the first storage area. When reading the second document identifier based on the storage address of the second document identifier, load all data in the physical storage unit where the second document identifier is located into the second storage area. If the inverted index of the target term is not fully loaded into the second storage area, update the second document identifier based on the document jump step size to load the inverted index of the target term into the second storage area.

4. The method according to claim 1, characterized in that, The method further includes: After treating each term in the target term list as a target term, the inverted index of each term in the target term list is loaded into the second storage area to provide target retrieval services based on the inverted index of each term in the target term list stored in the second storage area.

5. The method according to claim 1 or 4, characterized in that, The method further includes: When at least one term is obtained from the query request, the inverted index of each term in the query request is retrieved from the second storage area; If an inverted index for each term in the query request is found in the second storage area, document retrieval is performed based on the inverted index for each term in the query request, so as to determine the retrieval result of the query request based on the retrieved documents.

6. A data processing method, characterized in that, The method is applied to a data processing system, which includes a data access node and a data retrieval node. The method is executed by the data access node, and the method includes: Multiple threads are created for term counting; each of the multiple threads maintains a buffer area; the multiple threads include a target thread; When the target thread obtains at least one term parsed from the target query request, it writes the count value of each term in the target query request into the target buffer area maintained by the target thread. When the aggregation condition of the target buffer region is met, the count value of each term in the target buffer region is aggregated to obtain the aggregated count value of at least one candidate term. Write the aggregated count value of at least one candidate term into a global counting structure, statistically obtain at least one term that meets the popularity condition based on the global counting structure, and determine the target term list based on at least one term that meets the popularity condition. The target term list is stored in a data storage object associated with the data access node, so that the data retrieval node can obtain the target term list from the data storage object.

7. The method according to claim 6, characterized in that, The aggregation conditions include one or more of the following: the number of counts in the target buffer area reaches the capacity threshold of the target buffer area, and the current time reaches the preset aggregation time.

8. The method according to claim 6, characterized in that, The process of aggregating the count values ​​of each term in the target buffer region to obtain the aggregated count value of at least one candidate term includes: Merge identical terms in the target buffer region to obtain at least one candidate term, and sum the count values ​​corresponding to each candidate term to obtain the aggregate count value of each candidate term.

9. The method according to claim 6, characterized in that, The step of writing the aggregated count value of at least one candidate term into a global counting structure, and statistically obtaining at least one term that satisfies the popularity condition based on the global counting structure, includes: Obtain the aggregate count value of the at least one candidate term; the at least one candidate term includes the target candidate term; Based on the hash value corresponding to the target candidate term, a target counting segment for counting the target candidate term is determined from at least one counting segment in the global counting structure; the target counting segment includes the global count value corresponding to at least one candidate term; The aggregated count value of the target candidate term is used to update the global count value of the candidate terms recorded in the target count segment; When the hot word statistics condition of the global counting structure is met, at least one word that meets the heat condition is determined based on the global count value of the candidate words recorded in the at least one counting segment.

10. The method according to claim 9, characterized in that, The step of writing the aggregated count value of the target candidate term into the target count shard, and obtaining the global count value of the target candidate term through the target count shard, includes: If the term identifier corresponding to the target candidate term is not found in the target counting segment, then the global count value of each historical candidate term stored in the target counting segment is obtained, and the decay probability of each historical candidate term is calculated based on the global count value of each historical candidate term. The global count value of each historical candidate word is updated based on the decay probability of each historical candidate word. If there is a target historical candidate word in the target counting segment whose updated global count value is less than or equal to a preset value, then the target historical candidate word is replaced with the target candidate word, and the aggregated count value of the target candidate word is used as the global count value of the target candidate word.

11. The method according to claim 9, characterized in that, The step of writing the aggregated count value of the target candidate term into the target count shard, and obtaining the global count value of the target candidate term through the target count shard, includes: If the term identifier corresponding to the target candidate term is found in the target counting segment, the global count value of the target candidate term is updated based on the aggregate count value of the target candidate term.

12. A data processing apparatus, characterized in that, The device is applied to a data processing system, which includes a data access node and a data retrieval node. The device operates at the data retrieval node, and the device includes: The list retrieval unit is configured to retrieve a list of target terms from a data storage object associated with the data access node, and store an inverted index of each term in the list of target terms in a first storage area; the list of target terms is at least one term that satisfies a popularity condition as determined by the data access node; the at least one term includes the target term. The step size acquisition unit is used to search for the inverted chain of the target term in the second storage area of ​​the data retrieval node, and when the inverted chain of the target term is not found, to acquire the document jump step size that satisfies the jump constraint condition; the document jump step size refers to the number of document identifiers skipped each time the document identifiers in the inverted chain of the target term are read from the first storage area. The skip addressing unit is used to perform document skip addressing in multiple physical storage units included in the first storage area based on the storage address of the document identifier in the inverted list of the target term and the document skip step size, and load the inverted list of the target term into the second storage area based on the addressed physical storage unit, so as to provide data retrieval services based on the inverted list in the second storage area.

13. A data processing apparatus, characterized in that, The device is applied to a data processing system, which includes a data access node and a data retrieval node. The device operates at the data access node, and the device includes: A thread creation unit is used to create multiple threads for term counting; each of the multiple threads maintains a buffer area; the multiple threads include a target thread; A thread counting unit is used to write the count value of each term in the target query request into the target buffer area maintained by the target thread when the target thread obtains at least one term parsed from the target query request. An aggregation counting unit is used to aggregate the count value of each term in the target buffer region when the aggregation condition of the target buffer region is met, so as to obtain the aggregation count value of at least one candidate term. A global counting unit is used to write the aggregated count value of at least one candidate term into a global counting structure, statistically obtain at least one term that meets the popularity condition based on the global counting structure, and determine a target term list based on at least one term that meets the popularity condition. The list storage unit is used to store the target term list in a data storage object related to the data access node, so that the data retrieval node can obtain the target term list from the data storage object.

14. A computer device, characterized in that, Including memory and processor; The memory is connected to the processor, the memory is used to store computer programs, and the processor is used to invoke the computer programs so that the computer device performs the method according to any one of claims 1-11.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded and executed by a processor to cause a computer device having the processor to perform the method of any one of claims 1-11.

16. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium, the computer program being adapted to be read and executed by a processor to cause a computer device having the processor to perform the method of any one of claims 1-11.

Images