Index generation method, device, storage medium and electronic device

Abstract

The embodiments of the present application disclose an index generation method, device, storage medium and electronic device. The method includes: when a preset index generation operation request is obtained, determining a first position, the first position points to the position of the tail message corresponding to the preset index generation operation in the target message sequence; the target message sequence is used to store messages assigned to the target node; based on the first position, the preset index generation operation is performed to obtain a target index; determining a second position, the second position points to the position of the tail data in the target message sequence after the preset index generation operation is completed; constructing at least two threads based on the message extraction time parameter, message processing time parameter and number of CPU cores corresponding to the target node, the at least two threads are used to concurrently update the messages in the target message sequence that are located after the first position to the target index. The present application improves the speed of tracing back incremental messages.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to an index generation method, device, storage medium, and electronic device. Background Art

[0002] With the development of artificial intelligence and data search technologies, the volume of many businesses continues to grow. Data services that support large data volumes through indexing have gained widespread application in many business scenarios. To support large data volumes, related technologies can improve data service efficiency by building indexes. However, as data volumes increase, the time required to build indexes also increases. If indexing takes a long time, it will significantly impact the data service experience. However, the indexing speed of current related technologies is insufficient to meet the needs of large data services. Summary of the Invention

[0003] In order to solve at least one of the above technical problems, embodiments of the present application provide an index generation method, device, storage medium, and electronic device.

[0004] In one aspect, an embodiment of the present application provides an index generation method, which is applied to a target node, where the target node is any node in a distributed recall system. The method includes:

[0005] In the case of obtaining a preset index generation operation request, determining a first position, where the first position points to a position of a tail message corresponding to the preset index generation operation in a target message sequence; the target message sequence is used to store messages obtained by the distributed recall system and assigned to the target node for processing;

[0006] Executing the preset index generation operation based on the first position to obtain a target index, where the target index is an index corresponding to the distributed recall system;

[0007] Determine a second position, where the second position points to a position where the tail data in the target message sequence is located after the preset index generation operation is completed;

[0008] When the distance between the second position and the first position meets the first distance condition, at least two threads are constructed based on the message extraction time parameters, message processing time parameters and number of central processing unit cores corresponding to the target nodes respectively. The at least two threads are used to concurrently process the messages located after the first position in the target message sequence, and update the processing results to the target index.

[0009] On the other hand, an embodiment of the present application provides an index generation device, which is applied to a target node, where the target node is any node in a distributed recall system, and the device includes:

[0010] a first position determination module configured to, upon receiving a preset index generation operation request, determine a first position, wherein the first position refers to a position of a tail message corresponding to the preset index generation operation in a target message sequence; the target message sequence is configured to store messages obtained by the distributed recall system and assigned to the target node for processing;

[0011] A preset index generation module, configured to perform the preset index generation operation based on the first position to obtain a target index, where the target index is an index corresponding to the distributed recall system;

[0012] A second position determining module, configured to determine a second position, where the second position points to a position where the tail data in the target message sequence is located after the preset index generation operation is completed;

[0013] A message retrieval module is used to construct at least two threads based on the message extraction time parameters, message processing time parameters and number of central processing unit cores corresponding to the target nodes when the distance between the second position and the first position meets the first distance condition. The at least two threads are used to concurrently process the messages located after the first position in the target message sequence and update the processing results to the target index.

[0014] On the other hand, an embodiment of the present application provides a distributed recall system, which includes multiple nodes, and any node can implement the above-mentioned index generation method.

[0015] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above-mentioned index generation method.

[0016] On the other hand, an embodiment of the present application provides an electronic device comprising at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the above-mentioned index generation method by executing the instructions stored in the memory.

[0017] On the other hand, an embodiment of the present application provides a computer program product, including a computer program or instructions, which implements the above-mentioned index generation method when executed by a processor.

[0018] The embodiments of the present application provide an index generation method, device, storage medium, and electronic device. The index generation method is applied to the target node in the distributed recall system. For the target node, concurrent incremental message processing can be performed during the execution of the index generation operation by setting concurrent threads, making full use of the resources in the central processing unit and memory during the execution of the index generation operation, achieving the purpose of rapid retrieval of incremental messages, and significantly shortening the time required to obtain the final index. Ultimately, the distributed recall system running based on the index in each node can quickly reach the latest optimal index state, and provide services to the outside world as soon as possible based on the index in this state. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or related technologies, the following is a brief introduction to the drawings required for use in the embodiments or related technology descriptions. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 This is a schematic diagram of a feasible implementation framework of the index generation method provided in the embodiments of this specification;

[0021] Figure 2 This is a flowchart of an index generation method provided in an embodiment of the present application;

[0022] Figure 3 Schematic diagram of a concurrent index generation method provided in an embodiment of the present application;

[0023] Figure 4 This is a schematic diagram of a single-threaded message processing method provided by an embodiment of the present application;

[0024] Figure 5 This is a schematic diagram of a related art incremental message tracing method provided by an embodiment of the present application;

[0025] Figure 6 is a block diagram of an index generation device provided in an embodiment of the present application;

[0026] Figure 7 This is a schematic diagram of the hardware structure of a device provided in an embodiment of the present application for implementing the method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present application, not all of them. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the embodiments of the present application.

[0028] It should be noted that the terms "first", "second", etc. in the description and claims of the embodiments of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] In order to make the purpose, technical solutions and advantages disclosed in the embodiments of the present application more clearly understood, the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the embodiments of the present application and are not intended to limit the embodiments of the present application.

[0030] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "multiple" means two or more. In order to facilitate understanding of the above-mentioned technical solutions and the technical effects produced by the embodiments of this application, the embodiments of this application first explain the relevant professional terms:

[0031] Cloud technology: refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or local area network to achieve data computing, storage, processing and sharing. It is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool that can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The backend services of technical network systems require a large amount of computing and storage resources, such as video websites, image websites and more portals. With the rapid development and application of the Internet industry, every item may have its own identification mark in the future, and all of them need to be transmitted to the backend system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system support, which can only be achieved through cloud computing.

[0032] Intelligent Traffic System (ITS), also known as Intelligent Transportation System, effectively integrates advanced science and technology (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) into transportation, service control and vehicle manufacturing, strengthens the connection between vehicles, roads and users, and thus forms a comprehensive transportation system that ensures safety, improves efficiency, improves the environment and saves energy.

[0033] Intelligent Vehicle Infrastructure Cooperative Systems (IVICS), also known as VICS, are a development direction of Intelligent Transportation Systems (ITS). IVICS utilizes advanced wireless communications and next-generation internet technologies to implement dynamic, real-time information exchange between vehicles and roads. Based on the collection and integration of dynamic traffic information across time and space, IVICS conducts active vehicle safety control and collaborative road management. This fully realizes effective coordination between people, vehicles, and roads, ensuring traffic safety and improving traffic efficiency, resulting in a safe, efficient, and environmentally friendly road transportation system.

[0034] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0035] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0036] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0037] Deep learning: The concept of deep learning originates from the study of artificial neural networks. A multilayer perceptron with multiple hidden layers is an example of a deep learning architecture. Deep learning discovers distributed feature representations of data by combining lower-level features to form more abstract higher-level representations of attribute categories or features.

[0038] ElasticSearch is a distributed search and analytics engine. It provides near-real-time search and analytics for all types of data. Whether it's structured or unstructured text, numeric data, or geospatial data, Elasticsearch efficiently stores and indexes it in a way that supports fast searches. This allows users to go beyond simple data retrieval and aggregation to discover trends and patterns in their data. As your data and query volumes grow, Elasticsearch's distributed nature allows your deployment to scale seamlessly.

[0039] Redis, the full name of which is Remote Dictionary Server, is an open-source, network-enabled, log-based, key-value database that can be both memory-based and persistent, and provides application programming interfaces in multiple languages.

[0040] Kafka is an open-source stream processing platform. It's a high-throughput distributed publish-subscribe messaging system that handles streaming data from all consumer actions on websites. These actions (web browsing, searches, and other user actions) are a key factor in many current applications and services. This data is often processed through log processing and log aggregation due to throughput requirements. Kafka's purpose is to unify online and offline message processing and to provide real-time messaging across clusters.

[0041] With the development of artificial intelligence and data search technologies, the volume of many businesses continues to grow. Data services that support large data volumes through indexing have gained widespread application in many business scenarios. For example, in data search and content recommendation scenarios, there is a need to query data related to target information based on that target information. Systems that support these queries can be used as recall systems. To improve the efficiency of recall systems, indexes can be built for the data used in these scenarios.

[0042] The recall system can be distributed or non-distributed. Taking a distributed recall system as an example, the distributed recall system can be used as a search engine or a recommendation engine. This type of recall system can improve execution efficiency by building a preset index. In the embodiment of the present application, the preset index is taken as an example of a full index to describe the relevant technology and the technical solution in the embodiment of the present application. It can be understood that the preset index is not limited to the full index, and can also be used to build an index for part of the data in the entire data.

[0043] For full indexing, you can build the full index regularly or irregularly. For example, a distributed recall system used in a recommendation scenario could start building the full index at the end of each weekend to optimize its index structure and provide optimal support for the next week's service. Of course, the frequency of full indexing can be adjusted as the business volume of the recommendation scenario fluctuates.

[0044] During the full index generation process, users may still access the recall system, causing the recall system to cache incremental messages. The incremental messages refer to messages generated based on user access operations during the full index generation operation, or messages from upstream components of the recall system during the full index generation operation. Incremental messages also involve changes in data, so incremental messages should also be reflected in the final constructed full index. Therefore, the final constructed full index is not only based on the data before the full index generation operation, but also based on incremental messages. In other words, the above-mentioned full build operation refers to index generation of the full data before the full build operation, but the final constructed full index should reflect the information in the data before the full index generation operation and the information in the incremental messages during the full index generation operation.

[0045] In order to ensure that the final constructed full index can reflect the information in the data before the full index generation operation and the information in the incremental messages during the full index generation operation, the incremental messages generated during the full index generation process can be cached, and then after the full index generation operation is completed, the incremental messages can be traced back. Tracing back refers to the process of reflecting the information in the incremental messages in the final constructed full index.

[0046] When full index generation takes a relatively long time, message systems such as Kafka may cache a large number of incremental messages. How quickly nodes in a distributed recall system can process these incremental messages is crucial for shortening the index generation cycle. Currently, there are two main technical solutions to this problem:

[0047] The first type relies on open source components to solve the index generation problem. Distributed recall systems can be built based on ElasticSearch or Redis. This type of technical solution does not require users to generate full indexes themselves, nor does it require caching incremental messages. Instead, it directly handles data updates based on the interfaces provided by ElasticSearch or Redis, generating indexes that reflect the data updates. This type of technical solution does not have the problem of full index generation in design, but has execution efficiency bottlenecks and is also limited by the performance of ElasticSearch or Redis itself, leaving limited room for improvement.

[0048] The second type of solution does not rely on open source components to solve the index generation problem. This involves caching incremental messages during the full index generation process. After the full index generation is complete, a single-threaded approach is used to pull the cached incremental messages. These incremental messages are then processed and the results are updated to the index generated by the full index generation using a single thread. However, due to the low efficiency and slow speed of single-threaded message pulling and processing, the retrieval of incremental messages is slow, and obtaining the final full index takes a long time.

[0049] Of course, for the related technologies for building indexes for partial data, there are still the above-mentioned technical problems of slow incremental message retrieval speed and long time consumption to obtain the final preset index. In summary, the embodiment of the present application proposes that the technical solution for incremental message retrieval based on a single thread in the related technology is less efficient, and does not fully utilize the resources in the central processing unit and memory during the execution of the preset index generation operation, resulting in a long incremental message retrieval time, which leads to a long time consumption to obtain the final index. In view of this, the embodiment of the present application proposes an index generation method, which is applied to the target node in the distributed recall system. For the target node, concurrent incremental message processing can be performed during the execution of the index generation operation by setting concurrent threads, making full use of the resources in the central processing unit and memory during the execution of the index generation operation, achieving the purpose of rapid retrieval of incremental messages, and significantly shortening the time consumed to obtain the final index. Ultimately, the distributed recall system running based on the index in each node can quickly reach the latest optimal state of the index, and provide services to the outside world as soon as possible based on the index in this state.

[0050] The embodiments of the present application can be applied to public cloud, private cloud or hybrid cloud scenarios. For example, the nodes in the distributed recall system in the present application and the indexes in the nodes can be stored in the above-mentioned public cloud, private cloud or hybrid cloud. A private cloud is a cloud infrastructure and software and hardware resources created within a firewall so that various departments within an organization or enterprise can share resources in the data center. A public cloud usually refers to a cloud that is provided by a third-party provider for users to use. A public cloud is generally available through the Internet and may be free or low-cost. The core attribute of a public cloud is shared resource services. There are many instances of this cloud that can provide services in today's entire open public network. A hybrid cloud integrates a public cloud and a private cloud, and is the main model and development direction of cloud computing in recent years. Private clouds are mainly for enterprise users. For security reasons, enterprises prefer to store data in private clouds, but at the same time hope to obtain computing resources from public clouds. In this case, hybrid clouds are increasingly being adopted. They mix and match public and private clouds to achieve the best results. This personalized solution achieves the goal of both saving money and ensuring security.

[0051] See also Figure 1 , Figure 1 This is a schematic diagram of a feasible implementation framework of the index generation method provided in the embodiment of this specification. Figure 1 As shown, the implementation framework may include at least a client 10 and an index generation processor 20, wherein the client 10 and the index generation processor 20 communicate via a network 30. The index generation processor 20 may be understood as a distributed recall system comprising multiple nodes, each of which may independently or collaboratively construct an index for the distributed recall system. The distributed recall system may provide the client 10 with various recall-based services, such as recommendation services and search services, based on the constructed index.

[0052] Specifically, for any node in the distributed recall system, an index can be constructed for the distributed recall system by performing the following operations. In the embodiment of the present application, the index of the distributed recall system is referred to as a target index:

[0053] In the case of obtaining a preset index generation operation request, determining a first position, the first position pointing to the position of the tail message corresponding to the preset index generation operation in the target message sequence; the target message sequence is used to store the messages obtained by the distributed recall system and assigned to the target node for processing;

[0054] Executing the preset index generation operation based on the first position to obtain a target index, where the target index is an index corresponding to the distributed recall system;

[0055] Determine a second position, where the second position points to a position where the tail data in the target message sequence is located after the preset index generation operation is completed;

[0056] When the distance between the above-mentioned second position and the above-mentioned first position meets the first distance condition, at least two threads are constructed based on the message extraction time parameters, message processing time parameters and the number of central processing unit cores corresponding to the above-mentioned target nodes respectively. The above-mentioned at least two threads are used to concurrently process the messages located after the above-mentioned first position in the above-mentioned target message sequence, and update the processing results to the above-mentioned target index.

[0057] The framework described above in this embodiment of the present invention can provide the index generation capabilities required for services in various scenarios, including but not limited to cloud technology, cloud gaming, cloud rendering, artificial intelligence, smart transportation, assisted driving, video media, smart communities, and instant messaging. Each component in this framework can be a terminal device or a server. Terminal devices include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart appliances, and in-vehicle terminals.

[0058] The following describes an index generation method according to an embodiment of the present application. Figure 2 A flow chart of an index generation method provided by an embodiment of the present application is shown, and the index generation method can be applied to a target node, which is any node in a distributed recall system. The embodiment of the present application provides method operation steps as described above in the embodiment or flowchart, but more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the order of many steps and does not represent the only execution order. When the actual system, terminal device or server product is executed, it can be executed in sequence or in parallel (for example, in a parallel processor or multi-threaded processing environment) according to the method shown in the embodiment or the accompanying drawings. The above method may include:

[0059] S101. When a preset index generation operation request is obtained, a first position is determined, where the first position points to the position of the tail message corresponding to the preset index generation operation in the target message sequence; the target message sequence is used to store messages obtained by the distributed recall system and assigned to the target node for processing.

[0060] The target node can be understood as any node in the distributed recall system. In other words, any node in the distributed recall system can have the ability to build an index for the distributed recall system. Each node in the distributed recall system can independently or collaboratively build an index for the distributed recall system. In the embodiment of the present application, the index corresponding to the distributed recall system is called the target index, and each node can provide recall-based services to the client based on the constructed target index.

[0061] In the embodiment of the present application, the preset index generation operation is specifically described as a full index generation operation as an example. It can be understood that the preset index generation operation can also be used to build an index for part of the data in the entire data. In this case, the "partial data" can be understood as the "full data" for which the preset index generation operation is performed. Therefore, the execution logic and the technical solution mentioned in the embodiment of the present application taking the full index generation operation as an example are based on the same inventive concept. This embodiment of the present application will not be described in detail. For the full index generation operation, for example, the distributed recall system includes N valid nodes, and the full messages in the distributed recall system are all stored in Kafka. The full messages in Kafka can be divided into N parts, and each node processes one of the messages, and updates the message processing results to the target index in the distributed recall system. In this way, the efficiency of index generation can be significantly improved.

[0062] In step S101, the target message sequence can be understood as the message sequence in which the messages assigned to the target node for processing in the distributed recall system are located. For example, the storage space in Kafka can be divided into N parts, namely partition_1 to partition_N. Taking the target node as the first node among the N nodes as an example, the messages in the storage space of partition_1 are sorted in the order of their storage time to form the target message sequence. When the preset index generation operation request is obtained, the target node records the first position {partition_1, offset_1}, and offset_1 represents the offset of the first position relative to the first address of partition_1. The first position is the last position of all valid messages in Partition_1 before the preset index generation operation is executed. It can also be understood that in the process of the target node executing the preset index generation operation, the data processed is the data from the first address to the position of offset_1 in partition_1, and the data after offset_1 will not be processed by the preset index generation operation.

[0063] For the N nodes in the distributed recall system, the message sequences corresponding to each node can be obtained. The messages in these message sequences are stored in partition_1, partition_2, ..., partition_n. When all N nodes participate in the full index generation, each node will record its corresponding first position, that is, {partition_1, offset_1}, {partition_2, offset_2}, ..., {partition_n, offset_n}.

[0064] S102. Execute the preset index generation operation based on the first position to obtain a target index, where the target index is an index corresponding to the distributed recall system.

[0065] The target node extracts and processes the messages at the first position and before the first position and in the target message sequence one by one, and obtains the target index according to the processing result. The target index is the index corresponding to the distributed recall system. Therefore, each node can update the target index so that the target index reflects the result of the message processing performed by each node. The embodiment of the present application does not limit the processing of the message. For example, if the message represents a deletion operation on a certain data, the processing result of the message is to delete the index record corresponding to the certain data in the target index. If the message represents an addition operation on the index record of a certain data, the processing result of the message is to add the index record corresponding to the certain data in the target index.

[0066] S103. Determine a second position, where the second position points to the position where the tail data in the target message sequence is located after the preset index generation operation is completed.

[0067] However, the execution of the preset index generation operation is time-consuming. During the execution of the preset index generation operation, the target node may still provide services to the outside world, may obtain messages from the client with which it communicates, and may also obtain messages from the upstream components of the distributed recall system. These messages can still be assigned to the above-mentioned target message sequence, thereby causing the number of messages in the target message sequence to continue to increase. These incremental messages will not be processed by the preset index generation operation.

[0068] After the preset index generation operation is completed, the position where the last valid data in the target message sequence is located is the second position. Obviously, if the second position is close to the first position, it means that there are fewer incremental messages during the execution of the preset index generation operation. If the second position is far from the first position, it means that there are more incremental messages during the execution of the preset index generation operation.

[0069] For the N nodes in the distributed recall system, the second position corresponding to each node can be obtained, namely {partition_1, offset_now_1},..., {partition_n, offset_now_n}, where offset_now_1 represents the position of the tail data in the above target message sequence after the preset index generation operation of the target node is completed, and {partition_n, offset_now_n} represents the position of the tail data in the message sequence corresponding to the Nth node after the preset index generation operation is completed.

[0070] S104. When the distance between the second position and the first position satisfies the first distance condition, at least two threads are constructed based on the message extraction time parameters, message processing time parameters and the number of CPU cores corresponding to the target nodes respectively. The at least two threads are used to concurrently process the messages located after the first position in the target message sequence, and update the processing results to the target index.

[0071] The embodiment of the present application does not limit the first distance condition. For example, a distance threshold M can be set. If the distance between the second position and the first position is greater than M, it is determined that the first distance condition is met. Of course, M is an integer greater than or equal to 0. The value of M can be determined according to actual conditions, and the embodiment of the present application does not limit this.

[0072] If the distance between the second position and the first position satisfies the first distance condition, it can be considered that there are many incremental messages. In this case, the incremental messages can be quickly retrieved through concurrent message processing. Of course, if the distance between the second position and the first position does not meet the first distance condition, it can be considered that there are few incremental messages. In this case, it is also possible not to quickly retrieve the incremental messages through concurrent message processing. For example, the messages can be retrieved through a single thread. This embodiment of the present application does not limit this.

[0073] Specifically, the embodiment of the present application determines the total number of concurrent threads of the target node based on the message extraction time parameter, the message processing time parameter and the number of CPU cores; and constructs the at least two threads based on the total number of concurrent threads.

[0074] The embodiment of the present application does not limit the method for determining the total number of concurrent threads of the target node based on the above-mentioned message extraction time parameter, the above-mentioned message processing time parameter and the above-mentioned number of CPU cores. It can be set according to the actual situation, as long as the indicative role of these three parameters in determining the total number of concurrent threads is taken into account. Among them, the message extraction time parameter represents the average time it takes for the thread to extract the message from the target message sequence, and the above-mentioned message processing time parameter represents the time it takes for the thread to process the message and update the processing result to the target index. These three parameters comprehensively characterize the relevant performance and resources in incremental message tracing to reasonably determine the total number of concurrent threads.

[0075] In one embodiment, the above-mentioned determination of the total number of concurrent threads of the target node based on the message extraction time parameter, the message processing time parameter and the number of central processing unit cores includes: determining a first target value based on the ratio of the message extraction time parameter and the message processing time parameter; determining a second target value based on the reciprocal of the number of central processing unit cores; when the first target value is less than the second target value, rounding up the reciprocal of the second target value to obtain the total number of concurrent threads; when the first target value is greater than or equal to the second target value, rounding up the reciprocal of the first target value to obtain the total number of concurrent threads.

[0076] Specifically, the average time x (message extraction time parameter) for a single thread in the target node to pull a message from the target message sequence and the average time y (message processing time parameter) for a single thread in the target node to process a message and update the processing result in the target index (IndexDB) can be tested. The ratio x / y of these two time periods is the first target value. The second target value can be represented as 1 / CoreNum, where CoreNum is the number of CPU cores of the target node. Under normal circumstances, the value of x is generally relatively small. If the first target value is less than the second target value, then when the first target value is less than the second target value, the reciprocal of the second target value is rounded up to obtain the total number of concurrent threads m; when the first target value is greater than or equal to the second target value, the reciprocal of the first target value is rounded up to obtain the total number of concurrent threads m.

[0077] The index generation method provided by the embodiment of the present application is applied to any node in the distributed recall system, and any node can perform full index generation and incremental message tracing based on its own corresponding message sequence, so as to achieve the effect of each node collaboratively performing index generation and improve the efficiency of index generation. Moreover, for the target node, in the process of performing incremental message tracing, the target node can construct at least two threads based on the message extraction time-consuming parameters, message processing time-consuming parameters and the number of CPU cores corresponding to the above-mentioned target nodes, and perform message tracing concurrently. The concurrent threads take into account the message extraction time-consuming parameters, message processing time-consuming parameters and the number of CPU cores of the node, that is, fully take into account the CPU resource performance of the target node and the data processing performance of the target node, and improve the concurrency capability as much as possible, so as to achieve the purpose of rational resource utilization and efficient incremental message tracing while further improving the efficiency of index generation. Ultimately, the distributed recall system running based on the index in each node can quickly reach the latest optimal state of the index, and provide services to the outside world as soon as possible based on the index in this state.

[0078] In one embodiment, the concurrent processing of the messages located after the first position in the target message sequence includes:

[0079] S201. Construct a message queue corresponding to each of the above threads.

[0080] In the embodiment of this application, each concurrent thread has a corresponding message queue. Please refer to Figure 3 , which shows a schematic diagram of a concurrent index generation method in an embodiment of the present application. This method flowchart illustrates a scenario in which nodes in a distributed recall system collaborate to execute this concurrent index generation method. For example, partition_1 corresponds to the target node. The messages in partition_1 form the target message sequence. Based on the previous text, the total number of concurrent threads, m, can be determined. A queue is set for each thread, resulting in m queues: queue_1...queue_m.

[0081] S202. Extract the messages located after the first position in the target message sequence in sequence, and assign the extracted messages to the corresponding message queues through a global order-preserving mapping operation. The global order-preserving mapping operation is used to assign messages with the same identifier to the same message queue.

[0082] In the embodiment of the present application, step S202 can be implemented by setting a special thread. The specially set thread can be considered as a message pulling thread. Figure 3For example, for the target node, thread p1 is set to assign the message located after the first position in the target message sequence to the message queue. This application does not limit how to implement the mapping operation based on global order preservation, as long as the purpose of assigning messages with the same identifier to the same message queue is achieved. In the embodiment of the present application, global order preservation means that for any two messages with the same identifier, message 1 and message 2, if message 1 is generated earlier than message 2, and the message 1 and message 2 are located after the above-mentioned first position, then the message 1 and message 2 will be assigned to the same message queue, and message 1 will still be before message 2 in the message queue. The role of global order preservation is to ensure that the message processing timing is consistent with the message generation timing, thereby ensuring that the updated target index is correct.

[0083] In a specific embodiment, the extracted message can be hashed, and after performing a modulo operation based on m on the hash result, the message is assigned to the message queue corresponding to the modulo operation result. Figure 3 For example, the target node has a total of m concurrent threads and m message queues. The extracted messages are hashed and the calculation results are modulo-ed by m to obtain the modulo result n, which must be between 1 and m. Assigning the message to queue_n can achieve the global order-preserving mapping purpose.

[0084] In the embodiment of the present application, the capacity L of each message queue can also be set, and the product of L and m is less than a preset parameter. The preset parameter is set based on the memory of the target node to prevent memory overflow and achieve the purpose of fully utilizing the storage space to improve concurrency capabilities. The embodiment of the present application does not limit the value of the preset parameter. For example, it can be set not to exceed 80% of the remaining available storage space of the target node.

[0085] S203. Trigger each of the above threads to concurrently process the messages in the corresponding message queues.

[0086] Still Figure 3 For example, each thread extracts the messages in its corresponding message queue for processing without affecting each other. Of course, in some embodiments, the messages in the thread queue of the target node can also be merged based on global order preservation to form a hash bucket, and each concurrently executed thread extracts and processes the messages in the hash bucket.

[0087] Combine Figure 3 It can be seen that the speed of tracing back incremental messages is improved by processing the corresponding message sequences by multiple nodes respectively. For each node, concurrent multiple threads further improve the speed of tracing back incremental messages. Figure 3It can be seen that n*m threads can concurrently process messages in Kafka. This bipolar concurrent design can significantly improve the speed of tracing incremental messages.

[0088] In one embodiment, when there is a target message queue in each of the above-mentioned message queues, the above-mentioned operation of sequentially extracting the messages located after the above-mentioned first position in the above-mentioned target message sequence is stopped, and the above-mentioned target message queue is a message queue whose number of messages reaches a preset requirement; when the stopping time reaches a preset time threshold, the above-mentioned operation of sequentially extracting the messages located after the above-mentioned first position in the above-mentioned target message sequence is continued.

[0089] The target message queue can be considered as a queue that is full or about to be full of messages. Figure 3 For example, if queue_1 is full or about to be full, thread p1 can suspend pulling messages to allow the messages in the m message queues to be further processed. After freeing up more queue space, it can resume pulling messages. The embodiment of the present application does not limit this preset requirement. The preset requirement can be that the number of messages reaches the upper limit of the capacity of the message queue, or that the number of messages reaches 95% of the capacity of the message queue. The embodiment of the present application does not require this preset time threshold, and it can be set according to actual conditions, for example, it can be set to 100 milliseconds.

[0090] In one embodiment, before concurrently processing the messages located after the first position in the target message sequence, the method further includes: the target node stops providing services to the client, and the client is an electronic device that interacts with the distributed recall system. Figure 3 For example, Figure 3 All N nodes in the system suspend external services and focus on tracing incremental messages. In actual scenarios, the number of incremental messages is limited, and Figure 3 The concurrency capability in the system is very strong. Therefore, the speed of tracing back incremental messages is very fast. Suspending external services will not have a big impact. It can also achieve the goal of obtaining the optimal index as soon as possible and resuming services as soon as possible.

[0091] In one embodiment, when the above-mentioned distributed recall system meets the following target conditions, the above-mentioned target node starts to provide services to the above-mentioned client; the above-mentioned target conditions are: for each node in the above-mentioned distributed recall system, the distance between the first target position corresponding to the above-mentioned node and the second target position corresponding to the above-mentioned node meets the second distance condition, the above-mentioned first target position points to the position of the message currently being processed in the message sequence corresponding to the above-mentioned node during the concurrent processing of the above-mentioned node, and the above-mentioned second target position points to the position of the current tail data of the above-mentioned message sequence corresponding to the above-mentioned node during the concurrent processing of the above-mentioned node.

[0092] The distance between the first target location and the second target location actually represents how many messages remain unprocessed for the corresponding node. If the second distance condition is met, it can be considered that the remaining unprocessed messages are relatively small. Of course, the embodiment of the present application does not limit the specific content of the second distance condition. For example, the second distance condition can be determined to be met when the distance is less than a preset value. The preset value can be set as needed and is not limited.

[0093] Still Figure 3 For example, Figure 3 When the number of messages remaining in each partition that have not been pulled by the corresponding pulling thread is relatively small, for example, when the distance between the first target position and the second target position in each partition is less than 100 message positions, it can be determined that the second distance condition is met, indicating that the incremental messages are about to be chased. In this case, each node can start the service and resume the service to the client. In other words, only when Figure 3 The service will be suspended only when the following conditions are met. As mentioned above, Figure 3 The situation exists for a very short time, so the solution of the embodiment of the present application has little impact on the service experience of the client.

[0094] In one embodiment, when the distributed recall system meets the following target conditions, the distributed recall system is further used to stop concurrent processing operations of other nodes except the target node, and the other nodes are also used to start providing services to the client.

[0095] In other words, when there are not many unprocessed messages left, concurrent processing operations are no longer needed. Each node can recycle the resources of concurrent threads and fully provide services to the outside world to ensure the client's service experience.

[0096] After the target node starts providing services to the client, the method further includes: performing single-threaded polling processing on unprocessed messages in each message sequence of the distributed recall system, and updating the processing results to the target index.

[0097] When there are not many unprocessed messages left, it is still necessary to continue processing the unprocessed messages. However, because the total number of unprocessed messages is small, it is no longer necessary for multiple nodes to perform concurrent processing. Only one node is left to perform concurrent message processing. For example, the target node in the embodiment of the present application continues to process the unprocessed messages in the above-mentioned distributed recall system and updates the target index. The other nodes can reclaim the resources of the concurrent threads.

[0098] Please refer to Figure 4 , which shows a schematic diagram of a single-threaded message processing method. The target node retains a single thread to pull messages from the message sequences of each node in the distributed recall system and distribute them to m message queues. Message processing is then performed by m concurrent threads. The value of m here is as described above and is not repeated here. Figure 3 and Figure 4 The difference is that in the former, each node only focuses on processing the messages in its own message sequence, while Figure 4 There is only one node in the system, which processes the unprocessed messages in the message sequence corresponding to each node. At this time, the size of each message queue can be set to 5% of the available memory of the node. Of course, it can also be set to other values. In the embodiment of this application, Figure 3 The message processing mode is called the extreme speed mode. Figure 4 The message processing method is called normal mode. The high-speed mode has a fast retrieval speed for incremental messages, but the service must be suspended. Normal service can be maintained in normal mode, but the message retrieval speed is slow. By switching between high-speed mode and normal mode, the purpose of quickly retrieval of incremental messages and rapid restoration of external services can be achieved. Ultimately, the distributed recall system can quickly obtain the optimal target index and quickly reach a serviceable state.

[0099] In order to contrast with the technical solution of the embodiment of the present application, the embodiment of the present application provides a related technology. In scenarios such as search, recommendation and advertising, the index in the distributed recall system is generally built in full on a regular basis. When the full index is generated, it is pushed to each node for loading. After loading, it is necessary to trace back and process the incremental messages during the full index generation. Only after the tracing is completed can the final target index be obtained. Please refer to Figure 5, which shows a schematic diagram of the incremental message retrieval method of the related technology. For the incremental messages that need to be retrieved after the preset index generation operation, these messages can all be stored in various partitions in Kafka, such as partiotion_1 to partiotion_n. The related technology can pull the incremental messages in each partition in turn through single-threaded polling, and then perform single-threaded incremental message processing, and update the index of the distributed recall system according to the processing results.

[0100] Will Figure 5 and Figure 3 By comparison, in the extreme speed mode, the speed of incremental message retrieval is theoretically equivalent to n*m times that of the related technology. Figure 4 and Figure 3 By comparison, under normal mode, the incremental message retrieval speed is theoretically equivalent to m times that of the related technology.

[0101] In actual business scenarios, the embodiments of this application Figure 5 The execution effect of the related technology in the test was tested. The single-threaded message pulling and processing mode in the related technology was adopted to process 81920 messages in 25.34 seconds. Figure 3 The execution effect of the technology of multi-node parallel processing of incremental messages in the medium-speed mode was tested. Specifically, 4 nodes were used, each node was assigned a thread to pull messages, and each node was set with 4 threads to process the pulled messages in parallel. It took 2.92s to process 81920 messages. Figure 4 The performance of the technology of pulling messages from a single node and processing incremental messages in parallel in the normal mode was tested. When the number of concurrent threads was 4, it took 14.63 seconds to process 81,920 messages. According to the test data, the time taken by the extreme speed mode mentioned in the embodiment of the present application is only 2.92 / 25.34=11.5% of the related technology, saving 88.5% of the time. At the same time, the time taken by the normal mode is 14.63 / 25.34=57.7% of the related technology, saving 42.3% of the time.

[0102] The embodiments of the present application provide an index generation method, device, storage medium, and electronic device. The index generation method is applied to the target node in the distributed recall system. For the target node, concurrent incremental message processing can be performed during the execution of the index generation operation by setting concurrent threads, making full use of the resources in the central processing unit and memory during the execution of the index generation operation, achieving the purpose of rapid retrieval of incremental messages, and significantly shortening the time required to obtain the final full index. In addition, by setting the extreme speed mode and the normal mode, the combination of the two modes can not only enable the distributed recall system to quickly reach the latest optimal index state, but also quickly provide services to the outside world.

[0103] Please refer to Figure 6 , which shows a block diagram of an index generation device in this embodiment, applied to a target node, where the target node is any node in a distributed recall system, and the device includes:

[0104] A first position determination module 101 is configured to, upon receiving a preset index generation operation request, determine a first position, wherein the first position refers to the position of the tail message corresponding to the preset index generation operation in a target message sequence; the target message sequence is used to store messages obtained by the distributed recall system and assigned to the target node for processing;

[0105] A preset index generating module 102 is configured to perform the preset index generating operation based on the first position to obtain a target index, where the target index is an index corresponding to the distributed recall system;

[0106] A second position determining module 103 is configured to determine a second position, where the second position refers to the position of the tail data in the target message sequence after the preset index generation operation is completed;

[0107] The message retrieval module 104 is used to construct at least two threads based on the message extraction time parameters, message processing time parameters and the number of central processing unit cores corresponding to the above-mentioned target nodes when the distance between the above-mentioned second position and the above-mentioned first position meets the first distance condition. The above-mentioned at least two threads are used to concurrently process the messages located after the above-mentioned first position in the above-mentioned target message sequence, and update the processing results to the above-mentioned target index.

[0108] In one embodiment, the message retrieval module 104 is configured to perform the following operations:

[0109] Construct a message queue corresponding to each of the above threads;

[0110] Sequentially extract the messages that follow the first position in the target message sequence, and assign the extracted messages to corresponding message queues through a global order-preserving mapping operation, wherein the global order-preserving mapping operation is used to assign messages with the same identifier to the same message queue;

[0111] The above threads are triggered to concurrently process the messages in the corresponding message queues.

[0112] In one embodiment, the message retrieval module 104 is configured to perform the following operations:

[0113] If a target message queue exists in each of the message queues, the operation of sequentially extracting messages located after the first position in the target message sequence is stopped, and the target message queue is a message queue containing a number of messages that reaches a preset requirement;

[0114] When the stop duration reaches the preset duration threshold, the operation of sequentially extracting the messages in the target message sequence that are located after the first position is continued.

[0115] In one embodiment, the message retrieval module 104 is configured to perform the following operations:

[0116] The target node stops providing services to the client, where the client is an electronic device that interacts with the distributed recall system.

[0117] In one embodiment, the message retrieval module 104 is configured to perform the following operations:

[0118] When the distributed recall system meets the following target conditions, the target node starts providing services to the client;

[0119] The target condition is that for each node in the distributed recall system, the distance between the first target position corresponding to the node and the second target position corresponding to the node meets the second distance condition.

[0120] The first target position refers to the position of the message currently being processed in the message sequence corresponding to the node during the concurrent processing of the node.

[0121] The second target position points to the position where the current tail data of the message sequence corresponding to the node is located during the concurrent processing of the node.

[0122] In one embodiment, when the distributed recall system meets the following target conditions, the distributed recall system is further used to stop concurrent processing operations of other nodes except the target node, and the other nodes are also used to start providing services to the client.

[0123] In one embodiment, the message retrieval module 104 is configured to perform the following operations:

[0124] Perform single-threaded polling processing on the unprocessed messages in each message sequence of the above distributed recall system, and update the processing results to the above target index.

[0125] In one embodiment, the message retrieval module 104 is configured to perform the following operations:

[0126] Determine the total number of concurrent threads of the target node based on the message extraction time parameter, the message processing time parameter, and the number of CPU cores;

[0127] The at least two threads are constructed based on the total number of concurrent threads.

[0128] In one embodiment, the message retrieval module 104 is configured to perform the following operations:

[0129] Determining a first target value based on a ratio of the message extraction time parameter to the message processing time parameter;

[0130] Determining a second target value based on the reciprocal of the number of CPU cores;

[0131] In the case where the first target value is less than the second target value, the reciprocal of the second target value is rounded up to obtain the total number of concurrent threads;

[0132] When the first target value is greater than or equal to the second target value, the reciprocal of the first target value is rounded up to obtain the total number of concurrent threads.

[0133] The device part and the method embodiment in the embodiment of this application are based on the same inventive concept and will not be described in detail here.

[0134] Further, Figure 7 A schematic diagram of the hardware structure of a device for implementing the method provided in the embodiment of the present application is shown. The above-mentioned device may participate in constituting or include the apparatus or system provided in the embodiment of the present application. Figure 7As shown, the device 10 may include one or more (illustrated as 102a, 102b, ..., 102n in the figure) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 7 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 7 More or fewer components than shown, or with Figure 7 Different configurations shown.

[0135] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the device 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0136] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the above-mentioned method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the above-mentioned index generation method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0137] The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by the communications provider of the device 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In one embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0138] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of device 10 (or mobile device).

[0139] It should be noted that the above-mentioned order of the embodiments of the present application is for descriptive purposes only and does not represent the superiority or inferiority of the embodiments. The above-mentioned embodiments of the present application are described in terms of specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0140] Each embodiment of the present application is described in a progressive manner. Similar portions between the embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the device and server embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For relevant portions, refer to the descriptions of the method embodiments.

[0141] On the other hand, an embodiment of the present application provides a distributed recall system, which includes multiple nodes, and any node can implement the above-mentioned index generation method.

[0142] On the other hand, an embodiment of the present application provides a computer program product, including a computer program or instructions, which implements the above-mentioned index generation method when executed by a processor.

[0143] On the other hand, an embodiment of the present application provides a computer-readable storage medium, in which at least one instruction or at least one program is stored. The at least one instruction or at least one program is loaded and executed by a processor to implement the above-mentioned index generation method.

[0144] The instructions in the storage medium may execute an index generation method, which is applied to a target node, where the target node is any node in the distributed recall system. The method includes:

[0145] In the case of obtaining a preset index generation operation request, determining a first position, the first position pointing to the position of the tail message corresponding to the preset index generation operation in the target message sequence; the target message sequence is used to store the messages obtained by the distributed recall system and assigned to the target node for processing;

[0146] Executing the preset index generation operation based on the first position to obtain a target index, where the target index is an index corresponding to the distributed recall system;

[0147] Determine a second position, where the second position points to a position where the tail data in the target message sequence is located after the preset index generation operation is completed;

[0148] When the distance between the above-mentioned second position and the above-mentioned first position meets the first distance condition, at least two threads are constructed based on the message extraction time parameters, message processing time parameters and the number of central processing unit cores corresponding to the above-mentioned target nodes respectively. The above-mentioned at least two threads are used to concurrently process the messages located after the above-mentioned first position in the above-mentioned target message sequence, and update the processing results to the above-mentioned target index.

[0149] In one embodiment, the method further includes performing the following operations based on the at least two threads:

[0150] Construct a message queue corresponding to each of the above threads;

[0151] Sequentially extract the messages that follow the first position in the target message sequence, and assign the extracted messages to corresponding message queues through a global order-preserving mapping operation, wherein the global order-preserving mapping operation is used to assign messages with the same identifier to the same message queue;

[0152] The above threads are triggered to concurrently process the messages in the corresponding message queues.

[0153] In one embodiment, the method further includes:

[0154] If a target message queue exists in each of the message queues, the operation of sequentially extracting messages located after the first position in the target message sequence is stopped, and the target message queue is a message queue containing a number of messages that reaches a preset requirement;

[0155] When the stop duration reaches the preset duration threshold, the operation of sequentially extracting the messages in the target message sequence that are located after the first position is continued.

[0156] In one embodiment, before concurrently processing the messages located after the first position in the target message sequence, the method further includes:

[0157] The target node stops providing services to the client, where the client is an electronic device that interacts with the distributed recall system.

[0158] In one embodiment, the method further includes:

[0159] When the distributed recall system meets the following target conditions, the target node starts providing services to the client;

[0160] The target condition is that for each node in the distributed recall system, the distance between the first target position corresponding to the node and the second target position corresponding to the node meets the second distance condition.

[0161] The first target position refers to the position of the message currently being processed in the message sequence corresponding to the node during the concurrent processing of the node.

[0162] The second target position points to the position where the current tail data of the message sequence corresponding to the node is located during the concurrent processing of the node.

[0163] In one embodiment, when the distributed recall system meets the following target conditions, the distributed recall system is further used to stop concurrent processing operations of other nodes except the target node, and the other nodes are also used to start providing services to the client.

[0164] In one embodiment, after the target node starts providing services to the client, the target node is further configured to perform the following operations:

[0165] Perform single-threaded polling processing on the unprocessed messages in each message sequence of the above distributed recall system, and update the processing results to the above target index.

[0166] In one embodiment, the at least two threads are constructed based on the message extraction time parameter, the message processing time parameter, and the number of CPU cores corresponding to the target nodes, including:

[0167] Determine the total number of concurrent threads of the target node based on the message extraction time parameter, the message processing time parameter, and the number of CPU cores;

[0168] The at least two threads are constructed based on the total number of concurrent threads.

[0169] In one embodiment, determining the total number of concurrent threads of the target node based on the message extraction time parameter, the message processing time parameter, and the number of CPU cores includes:

[0170] Determining a first target value based on a ratio of the message extraction time parameter to the message processing time parameter;

[0171] Determining a second target value based on the reciprocal of the number of CPU cores;

[0172] In the case where the first target value is less than the second target value, the reciprocal of the second target value is rounded up to obtain the total number of concurrent threads;

[0173] When the first target value is greater than or equal to the second target value, the reciprocal of the first target value is rounded up to obtain the total number of concurrent threads.

[0174] The above is only a preferred embodiment of the embodiment of the present application and is not intended to limit the embodiment of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the embodiment of the present application should be included in the scope of protection of the embodiment of the present application.

Claims

1. A method for generating an index, characterized in that: Applied to a target node, where the target node is any node in a distributed recall system, the method includes: In the case of obtaining a preset index generation operation request, determining a first position, where the first position points to a position of a tail message corresponding to the preset index generation operation in a target message sequence; the target message sequence is used to store messages obtained by the distributed recall system and assigned to the target node for processing; Executing the preset index generation operation based on the first position to obtain a target index, where the target index is an index corresponding to the distributed recall system; Determine a second position, where the second position points to a position where the tail data in the target message sequence is located after the preset index generation operation is completed; When the distance between the second position and the first position meets the first distance condition, at least two threads are constructed based on the message extraction time parameters, message processing time parameters and number of central processing unit cores corresponding to the target nodes respectively. The at least two threads are used to concurrently process the messages located after the first position in the target message sequence, and update the processing results to the target index.

2. The method according to claim 1, characterized in that The method further includes performing the following operations based on the at least two threads: Constructing a message queue corresponding to each of the threads; Sequentially extracting messages that are located after the first position in the target message sequence, and assigning the extracted messages to corresponding message queues through a global order-preserving mapping operation, wherein the global order-preserving mapping operation is used to assign messages with the same identifier to the same message queue; Each of the threads is triggered to concurrently process messages in the message queues corresponding to each of the threads.

3. The method according to claim 2, characterized in that The method further comprises: If a target message queue exists in each of the message queues, stopping the operation of sequentially extracting messages located after the first position in the target message sequence, the target message queue being a message queue containing a number of messages that reaches a preset requirement; When the stop duration reaches the preset duration threshold, the operation of sequentially extracting the messages in the target message sequence that are located after the first position is continued.

4. The method according to claim 2 or 3, characterized in that Before concurrently processing the messages located after the first position in the target message sequence, the method further includes: The target node stops providing services to the client, where the client is an electronic device that interacts with the distributed recall system.

5. The method according to claim 4, characterized in that The method further comprises: When the distributed recall system meets the following target conditions, the target node starts providing services to the client; The target condition is: for each node in the distributed recall system, the distance between the first target position corresponding to the node and the second target position corresponding to the node meets the second distance condition, The first target position refers to the position of the message currently being processed by the node in the message sequence corresponding to the node during the concurrent processing process. The second target position points to the position where the current tail data of the message sequence corresponding to the node is located during the concurrent processing of the node.

6. The method according to claim 5, characterized in that When the distributed recall system meets the following target conditions, the distributed recall system is further configured to stop concurrent processing operations of other nodes except the target node, and the other nodes are further configured to start providing services to the client.

7. The method according to claim 5 or 6, characterized in that After the target node starts providing services to the client, the target node is further configured to perform the following operations: Single-threaded polling processing is performed on the unprocessed messages in each message sequence of the distributed recall system, and the processing results are updated to the target index.

8. The method according to claim 1, characterized in that The constructing of at least two threads based on the message extraction time parameter, the message processing time parameter, and the number of CPU cores corresponding to the target nodes respectively includes: Determining the total number of concurrent threads of the target node based on the message extraction time parameter, the message processing time parameter, and the number of CPU cores; The at least two threads are constructed based on the total number of concurrent threads.

9. The method according to claim 8, characterized in that The determining the total number of concurrent threads of the target node based on the message extraction time parameter, the message processing time parameter, and the number of central processing unit cores includes: Determining a first target value based on a ratio of the message extraction time-consuming parameter to the message processing time-consuming parameter; Determining a second target value based on the reciprocal of the number of CPU cores; When the first target value is less than the second target value, rounding up the reciprocal of the second target value to obtain the total number of concurrent threads; When the first target value is greater than or equal to the second target value, the reciprocal of the first target value is rounded up to obtain the total number of concurrent threads.

10. An index generating device, characterized in that: Applied to a target node, the target node being any node in a distributed recall system, the device comprises: a first position determination module configured to, upon receiving a preset index generation operation request, determine a first position, wherein the first position refers to a position of a tail message corresponding to the preset index generation operation in a target message sequence; the target message sequence is configured to store messages obtained by the distributed recall system and assigned to the target node for processing; A preset index generation module, configured to perform the preset index generation operation based on the first position to obtain a target index, where the target index is an index corresponding to the distributed recall system; A second position determining module, configured to determine a second position, where the second position points to a position where the tail data in the target message sequence is located after the preset index generation operation is completed; A message retrieval module is used to construct at least two threads based on the message extraction time parameters, message processing time parameters and number of central processing unit cores corresponding to the target nodes when the distance between the second position and the first position meets the first distance condition. The at least two threads are used to concurrently process the messages located after the first position in the target message sequence and update the processing results to the target index.

11. A distributed recall system, characterized in that: The distributed recall system includes multiple nodes, and any node implements an index generation method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by a processor to implement an index generation method according to any one of claims 1 to 9.

13. An electronic device, characterized in that: It includes at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the at least one processor implements an index generation method as described in any one of claims 1 to 9 by executing the instructions stored in the memory.

14. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, an index generation method according to any one of claims 1 to 9 is implemented.

Images