Message data consumption method and device based on cache mechanism and multi-threading

Abstract

The present application provides a message data consumption method and device based on a cache mechanism and multi-threading, which performs task splitting on message data according to a preset number of threads to obtain multiple copies of task data, and distributes the multiple copies of task data to threads, splits the message data to multiple threads for processing, and improves the efficiency of data processing. In any thread, based on the judgment conditions preset in the data dictionary, the input task data is filtered to obtain quasi-linear data or non-class linear data. If quasi-linear data is obtained, the quasi-linear data is processed by business logic to obtain processed data, and the processed data is stored in a storage center; if non-class linear data is obtained, the non-class linear data is directly stored in a storage center; the data is distributed to different threads for asynchronous processing, and the received task data is compared with the data dictionary, and the processing of asynchronous quasi-linear data is realized at one time, thereby improving the data processing efficiency.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a message data consumption method and device based on a cache mechanism and multi-threading. Background Art

[0002] Big data refers to the massive and diverse transaction data, interaction data and sensor data that need to be quickly acquired, processed and analyzed to extract value from them. The scale often reaches the PB (1024TB) level. Different organizations have different definitions of big data. In the development of big data in recent years, distributed stream processing of data has dominated, and the most widely used is message queue (Mq).

[0003] Message refers to the data transmitted between applications. Messages can be very simple, such as only containing text strings, or more complex, possibly containing embedded objects. In terms of big data message transmission, message queues have become the mainstream application in the field of distributed topology data stream transmission. Among them, those who store data in the message queue are called producers, and those who obtain data from the message queue are called consumers. Message queues provide an efficient transmission mode for the transmission of massive data. At present, the main development direction of message queues is distributed, high throughput, and high scalability. Message queue middleware is an important component in distributed systems, mainly solving problems such as application coupling, asynchronous messages, and traffic clipping. Achieve high-performance, high-availability, scalable, and eventually consistent architecture. It is an indispensable middleware for large distributed systems.

[0004] Although the emergence of message queues provides asynchronous solutions for many scenarios, and significantly improves efficiency compared to previous synchronous processing, message queues cannot provide a suitable asynchronous processing solution for business scenarios that rely on data linearity (message queue business with time sequence). Summary of the invention

[0005] In view of this, the purpose of the present application is to propose a message data consumption method and device based on a cache mechanism and multi-threading to solve the above-mentioned problems.

[0006] Based on the above purpose, the first aspect of the present application provides a message data consumption method based on a cache mechanism and multi-threading, which is provided with a storage center for storing data and a data dictionary for filtering data, and the method comprises:

[0007] Pull message data linearly from the message queue based on a pre-set offset;

[0008] Splitting the message data into tasks based on a preset number of threads to obtain multiple copies of task data, and distributing the multiple copies of task data by threads;

[0009] For task data of any thread:

[0010] Based on the judgment conditions preset in the data dictionary, the input task data is filtered to obtain quasi-linear data or non-quasi-linear data;

[0011] In response to determining that the quasi-linear data is obtained, performing business logic processing on the quasi-linear data to obtain processed data, and storing the processed data in the storage center;

[0012] In response to determining that the non-linear data is obtained, storing the non-linear data in the storage center;

[0013] In response to determining that the total amount of the processed data and the non-linear data in the storage center is equal to the offset, data is again linearly pulled from the message queue based on the offset.

[0014] The second aspect of the present application provides a message data consumption device based on a cache mechanism and multi-threading, comprising:

[0015] The server is configured to: linearly pull message data from the message queue based on a pre-set offset;

[0016] Splitting the message data into tasks based on a preset number of threads to obtain multiple copies of task data, and distributing the multiple copies of task data by threads;

[0017] In response to determining that the total amount of the processed data and the non-linear data in the storage center is equal to the offset, linearly pulling data from the message queue again based on the offset;

[0018] The asynchronous processor is configured to: filter the input task data to obtain quasi-linear data or non-quasi-linear data based on the judgment conditions preset in the data dictionary;

[0019] In response to determining that the quasi-linear data is obtained, performing business logic processing on the quasi-linear data to obtain processed data, and storing the processed data in the storage center;

[0020] In response to determining that the non-linear data is obtained, the non-linear data is stored in the storage center.

[0021] The third aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method provided in the first aspect of the present application is implemented.

[0022] The fourth aspect of the present application provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to execute the method provided in the first aspect of the present application.

[0023] From the above description, it can be seen that the message data consumption method and device based on the cache mechanism and multi-thread provided by the present application, after linearly pulling the message data from the message queue based on the preset offset, splits the message data into tasks according to the preset number of threads to obtain multiple copies of task data, and distributes the multiple copies of task data to threads, and each thread simultaneously performs asynchronous processing on the distributed task data, splits the message data into multiple threads for processing, and improves the efficiency of data processing. In any thread, the input task data will be filtered based on the judgment conditions preset in the data dictionary to obtain linear data or non-linear data. If linear data is obtained, the linear data is processed by business logic to obtain processed data, and the processed data is stored in the storage center; if non-linear data is obtained, the non-linear data is directly stored in the storage center; the data is assigned to different threads for asynchronous processing, and the received task data is compared with the data dictionary, and the processing of asynchronous linear data is realized at one time, thereby improving the processing efficiency of the data. The processing data and non-linear data are stored in the storage center to check for omissions and fill in the gaps. When message data with data dictionary filtering errors occurs, the message data can be restored through the storage center. You can also design a scheduled task to perform a second check on the message data that has been filtered out in the storage center, thereby reducing the probability of incorrectly filtering data. If the total number of processed data and non-linear data in the storage center is equal to the offset, it means that the message data pulled this time has been processed. You can pull the message data linearly from the message queue again based on the offset for the next asynchronous processing of the message data. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 This is a flowchart of the synchronization process of the registration process of the embodiment of the present application;

[0026] Figure 2 This is a flowchart of the asynchronous processing of the registration process of an embodiment of the present application;

[0027] Figure 3 This is a flow chart of a message data consumption method based on a cache mechanism and multi-threading according to an embodiment of the present application;

[0028] Figure 4 This is a data dictionary filtering flow chart of an embodiment of the present application;

[0029] Figure 5 A flowchart of obtaining a second timestamp according to an embodiment of the present application;

[0030] Figure 6 This is a flowchart of index acquisition in an embodiment of the present application;

[0031] Figure 7 This is a schematic diagram of the structure of a message data consumption device based on a cache mechanism and multi-threading according to an embodiment of the present application;

[0032] Figure 8 This is a schematic diagram of a message data asynchronous processing process of a message data consumption device based on a cache mechanism and multi-threading according to an embodiment of the present application;

[0033] Fig. 9 A schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0035] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0036] It should be noted that, as described in the background technology, before the emergence of message queues, the commonly used message processing method is synchronous processing. Taking a user registration process as an example, the synchronous processing is explained. After the user successfully registers, the system needs to send a SMS registration success notification and give points for successful registration, such as Figure 1 As shown, the total time consumed for synchronization processing is 10ms+100ms+100ms=210ms. After the user registers and writes the message, it is necessary to wait for 10ms of user registration + 100ms of SMS notification + 100ms of increasing points before returning to the client.

[0037] However, since SMS notification and points increase are non-core processes, they can be transformed into asynchronous processing to improve system responsiveness. The emergence of message queues provides an asynchronous solution for this scenario, such as Figure 2 As shown in the figure, after SMS notification and points increase are changed to asynchronous form, the message is written after the user registers and the success is immediately returned to the client in about 10ms, without waiting for the long-time synchronization (SMS + points), which greatly improves the system throughput. Therefore, the typical scenario of asynchronous is to use the message queue to realize asynchronous operation that is time-consuming and does not require immediate (synchronous) return results.

[0038] Although the emergence of message queues provides asynchronous solutions for many scenarios, the efficiency has been significantly improved compared to previous synchronous processing. However, message queues cannot provide suitable asynchronous processing solutions for business scenarios that rely on linear characteristics of data (message queue business in time sequence). For example: in some business scenarios, the data transmitted through the message queue contains a timestamp, and the consumer's processing of some business data in the message after consumption is strongly dependent on the timestamp, and the maximum category of data contained in the message is determined, but the actual number of data categories contained is uncertain. In this case, in the current message queue design, the consumer's consumption and processing of messages must be synchronous and linear. At this time, the message processing speed depends entirely on the consumer's physical resources and server performance. When the server resources that the consumer relies on are limited, it will greatly affect the consumer's consumption of messages and reduce the efficiency of message processing.

[0039] This application addresses the problem that message queues cannot provide efficient message consumption and processing for linear consumers. It provides a message data consumption method based on a caching mechanism and multi-threading for message queue business system scenarios that rely on chronological order. Under the condition of server performance limitations, it can effectively improve the processing efficiency of message data.

[0040] In some embodiments, Figure 3As shown, a message data consumption method based on a cache mechanism and multi-threading is provided with a storage center for storing data and a data dictionary for filtering data. The method includes:

[0041] Step 100: linearly pull message data from the message queue based on a preset offset.

[0042] In this step, since there are a huge number of message data in the message queue, in order to avoid repeatedly pulling the same message data, the offset is set in advance according to the server performance and actual needs and other conditions, and the message data is linearly pulled from the message queue according to the offset. Linear pulling refers to pulling in a certain order and calibrating the position node of this pulling. Generally, numerical order is selected for linear pulling, that is, starting from a given starting point, continuously pulling the message data corresponding to the offset, and calibrating the position of the last message data in the message queue. The next message data pulling will be linearly pulled with the data after the last message data as the starting point. An example is as follows: if the preset offset is 2000, with the first message data as the starting point, the first pull will pull the first 2000 message data and mark the 2000th message data, the second pull will start from the 2001st message and end at the 4000th message data, and mark the 4000th message data, and so on, wherein 2000 messages are pulled simultaneously each time.

[0043] Step 200, splitting the message data into tasks based on a preset number of threads to obtain multiple copies of task data, and distributing the multiple copies of task data by threads.

[0044] In this step, the efficiency of processing a large amount of data by a single thread is low. According to the actual needs and the actual performance of the server, multi-threaded parallel asynchronous processing is set, wherein each thread corresponds to an asynchronous processor, which can speed up the processing speed of the message data pulled once, and the impact between multiple threads is small. Multi-threaded parallel processing is equivalent to increasing the processing speed by a multiple corresponding to the number of threads, shortening the processing time of the message data once. An example is given as follows: As shown in the example of step 100, 2000 message data are pulled at one time. If N threads are set, the message data is evenly split into 2000 / N parts, each corresponding to a task data. If it cannot be divided evenly, it is rounded up, but the number of threads is generally fixed, and the offset can be set manually before pulling the message data. Generally, a multiple of the number of threads is selected as the offset. After the message data is split, the 2000 / N parts of the split message data are simultaneously allocated to different threads. Optionally, the splitting process can also be linearly split, that is, split in sequence, starting from the first message data, and every 2000 / N data is divided into a group.

[0045] It should be noted that each thread processes the assigned task data in the same way, so the processing of task data of one thread is taken as an example for explanation. The processing process is as follows:

[0046] Step 300 , based on the judgment conditions preset in the data dictionary, the input task data is filtered to obtain quasi-linear data or non-quasi-linear data.

[0047] In this step, in order to ensure the efficiency of data dictionary reading, the present application stores the data dictionary in the cache to improve the efficiency of each data dictionary reading. When the task data is distributed, each message in the task data is judged one by one according to the judgment conditions preset in the data dictionary. If a message in the task data meets the requirements of the data dictionary, the message is judged as a quasi-linear message, and all quasi-linear messages in this thread constitute quasi-linear data; if a message in the task data does not meet the requirements of the data dictionary, the message is judged as a quasi-linear message, and all non-quasi-linear messages in this thread constitute non-quasi-linear data.

[0048] Optionally, before the data dictionary compares and filters the task data, the task data is cleaned of dirty data to prevent the dirty data in the task data from affecting the comparison results of the data dictionary. The filtering speed of the data dictionary on the task data can also be increased, thereby improving the processing efficiency of the entire message data.

[0049] Step 400, in response to determining that quasi-linear data is obtained, business logic processing is performed on the quasi-linear data to obtain processed data, and the processed data is stored in a storage center.

[0050] In this step, all the quasi-linear messages that meet the requirements of the data dictionary are screened out from the task data to form quasi-linear data, indicating that the quasi-linear messages in the quasi-linear data are valid data. The quasi-linear messages in the quasi-linear data can be selected to perform corresponding business logic processing to obtain processed data, and the processed data is stored in the storage center for reading or calling by the terminal device.

[0051] Step 500, in response to determining that non-linear data is obtained, the non-linear data is stored in a storage center.

[0052] In this step, all non-linear messages that do not meet the requirements of the data dictionary are screened out in the task data to form non-linear data, indicating that the non-linear messages in the non-linear data are expired and invalid data. The non-linear data cannot be used for corresponding business logic processing. The non-linear data is directly stored in the storage center for subsequent use.

[0053] Step 600, in response to determining that the total amount of processed data and non-linear data in the storage center is equal to the offset, linearly pull data from the message queue again based on the offset.

[0054] In this step, when the task data of all threads are asynchronously processed, that is, the total amount of processed data and non-linear data in the storage center is equal to the offset, as described in step 100, the server will continue to pull data according to the offset, that is, pull the 2001st to 4000th messages in the message queue for the next round of message data processing.

[0055] It should be noted that the asynchronous processing process corresponding to step 300 is an important part of the message data consumption method based on caching mechanism and multi-threading. In this link, the task data of the asynchronous processors corresponding to different threads are compared with the data dictionary to realize the asynchronous processing of the message data at one time, thereby improving the data processing efficiency.

[0056] In some embodiments, Figure 4 As shown, step 300: based on the judgment conditions preset in the data dictionary, obtain quasi-linear data or non-quasi-linear data according to the input task data, specifically including:

[0057] Step 310: extract the first timestamp in the task data.

[0058] In this step, the timestamp is a complete and verifiable data that can indicate that a piece of data already existed at a specific point in time. It is proposed mainly to provide users with an electronic evidence to prove the time when some of the user's data was generated. Each message in the task data has a generation time. In this step, the timestamp that proves the message needs to be extracted for subsequent use.

[0059] Step 320: Obtain a second timestamp in the data dictionary that corresponds to the first timestamp.

[0060] In this step, the second timestamp is used to filter out messages that do not meet the preset judgment conditions, so it is necessary to first determine the second timestamp corresponding to the first timestamp in the data dictionary. The message corresponding to the second timestamp and the message corresponding to the first timestamp are of the same message type. The situation of the message in the data of this task can be determined by comparing the time sequence of the first timestamp and the second timestamp.

[0061] Step 330 , in response to determining that the first timestamp is later than the second timestamp, the task data is filtered through a data dictionary to obtain quasi-linear data.

[0062] In this step, the moment corresponding to the first timestamp is later than the moment corresponding to the second timestamp in the data dictionary in the time series, indicating that the generation time of the message corresponding to the first timestamp is later than the generation time of the second timestamp record in the data dictionary, indicating that the message corresponding to the first timestamp is the latest generated and is a valid message. The message is marked as a quasi-linear message. All quasi-linear messages constitute quasi-linear data, which can be used to handle corresponding business.

[0063] Step 340 , in response to determining that the first timestamp is earlier than the second timestamp, the task data is filtered through a data dictionary to obtain quasi-linear data.

[0064] In this step, the moment corresponding to the first timestamp is earlier in the time series than the moment corresponding to the second timestamp in the data dictionary, indicating that the generation time of the message corresponding to the first timestamp is earlier than the generation time of the second timestamp record in the data dictionary, indicating that the message corresponding to the first timestamp is not the latest one, but an expired message, an invalid message, and the message is marked as a non-linear message. All non-linear messages constitute linear data. Since the non-linear data has expired, it cannot be used to handle the corresponding business and is directly stored in the storage center.

[0065] An example is given in conjunction with this embodiment, as shown below: Taking user registration as an example, if the generation time of the registration message in the task data is 13:00, and the generation time of the second timestamp record corresponding to the registration message in the data dictionary is 12:59, it means that the user entered the message data once for the registration service at 12:59, and then entered the data again for the same registration service at 13:00, which means that the data entered at 12:59 did not make the user register successfully. The user re-entered the data at 13:00, and the registration task is processed based on the latest data. If the generation time of the registration message in the task data is 12:58, it means that the registration message is an expired message, and it is directly stored in the storage center without executing the complete registration process, thereby improving the processing efficiency of message input data.

[0066] In some embodiments, Figure 5 As shown, step 320, obtaining a second timestamp corresponding to the first timestamp in the data dictionary, specifically includes:

[0067] Step 321: Determine the index corresponding to the first timestamp in the data dictionary.

[0068] In this step, the index in the data dictionary is a data type that is dependent on the timestamp, and the second timestamp can be found in the data dictionary according to the data type of the message from which the first timestamp is extracted.

[0069] Step 322: Based on the index, obtain a second timestamp corresponding to the index in the data dictionary.

[0070] In this step, since the index corresponds to the second timestamp one-to-one, after the index corresponding to the first timestamp is found, the second timestamp can be found according to the index.

[0071] In some embodiments, Figure 6 As shown, step 311, determining the index corresponding to the first timestamp in the data dictionary, specifically includes:

[0072] Step 3211: based on the first timestamp, extract the linear information field corresponding to the first timestamp from the task data.

[0073] Step 3212: Based on the linear information field, determine the data type corresponding to the linear information field.

[0074] Step 3213, obtain the index corresponding to the data type in the digital dictionary.

[0075] Among them, firstly, a linear information field that can determine the data type in the message corresponding to the first timestamp is determined, and the data type of the message is determined according to the linear field information, and then an index matching the data type is found in the data dictionary.

[0076] In some embodiments, before storing the processed data in the storage center, the method further includes: replacing the second timestamp in the data dictionary with the first timestamp.

[0077] Among them, when the first timestamp is later than the second timestamp, the message corresponding to the first timestamp is generated later. After the second timestamp in the data dictionary is replaced with the first timestamp, the first timestamp will replace the second timestamp to judge whether subsequent messages of the same type are expired.

[0078] In some embodiments, the message data consumption method based on the cache mechanism and multi-threading also includes: storing the replaced second timestamp in a storage center.

[0079] In response to determining that there is an error in the message data, the second timestamp of the storage center is retrieved to restore the message data.

[0080] The storage center is used to check for omissions and fill in the gaps. When data dictionary filters out incorrect data, data can be restored through the storage center. A scheduled task can also be designed to perform a second check on the filtered data in the storage center, thereby reducing the probability of incorrectly filtering data. Therefore, the replaced second timestamp does not need to be deleted immediately, but stored in the storage center for use during data recovery or a second check.

[0081] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0082] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0083] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a message data consumption device based on a cache mechanism and multi-threading.

[0084] refer to Figure 7 and Figure 8 , a message data consumption device based on a cache mechanism and multi-threading, including:

[0085] Server 1 is configured to: linearly pull message data from the message queue based on a preset offset;

[0086] The message data is split into tasks based on a preset number of threads to obtain multiple copies of task data, and the multiple copies of task data are distributed by threads;

[0087] In response to determining that the total amount of processed data and non-linear data in the storage center 4 is equal to the offset, linearly pulling data from the message queue again based on the offset;

[0088] The asynchronous processor 2 is configured to: filter the input task data to obtain quasi-linear data or non-quasi-linear data based on the judgment conditions preset in the data dictionary 3;

[0089] In response to determining that quasi-linear data is obtained, performing business logic processing on the quasi-linear data to obtain processed data, and storing the processed data in a storage center 4;

[0090] In response to determining that non-linear data is obtained, the non-linear data is stored in the storage center 4 .

[0091] The storage center 4 can be deployed on a local server or a remote server. The storage center 4 is configured to: when data with filtering errors in the data dictionary 3 appears, data recovery is performed through the storage center 4; based on a preset timed task, a secondary check is performed on the non-linear data filtered out in the storage center 4, thereby reducing the probability of incorrect filtering. In order to ensure the efficiency of reading the data dictionary 3, the present application stores the data dictionary 3 in a cache to improve the efficiency of reading the data dictionary 3 each time. The data dictionary 3 is configured to: store the timestamp corresponding to each data that depends on the timestamp when it is most recently processed.

[0092] Among them, the server 1 pulls the preset offset number of messages from the message queue as message data, and evenly splits the message data according to the number of asynchronous processors 2. The server 1 distributes the split task data to different asynchronous processors 2 at the same time. The asynchronous processor 2 performs preliminary cleaning on the task data and filters out the dirty data in the message data. Then the asynchronous processor 2 compares the first timestamp carried by each message in the task data with the second timestamp in the data dictionary 3. If the first timestamp is later than the second timestamp, it is determined that the message in the task data meets the requirements of the data dictionary 3 and is quasi-linear data. The data dictionary 3 replaces the stored second timestamp with the first timestamp. The server 1 can use the linear data to handle related business and store the data processed by the business logic in the storage center 4; if the first timestamp is earlier than the second timestamp, it is determined that the message in the task data does not meet the requirements of the data dictionary 3 and is non-quasi-linear data. These messages in the task data will be filtered out by the data dictionary 3 and directly stored in the storage center 4. When all asynchronous processors 2 complete the processing of their respective task data, the server 1 pulls the next message data according to the offset.

[0093] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0094] The device of the above embodiment is used to implement the corresponding message data consumption method based on cache mechanism and multi-threading in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0095] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the message data consumption method based on the cache mechanism and multi-threading described in any of the above embodiments is implemented.

[0096] Fig. 9A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0097] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0098] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0099] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0100] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0101] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0102] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0103] The electronic device of the above embodiment is used to implement the corresponding message data consumption method based on cache mechanism and multi-threading in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0104] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the message data consumption method based on the cache mechanism and multi-threading as described in any of the above embodiments.

[0105] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0106] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the message data consumption method based on the cache mechanism and multi-threading as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0107] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0108] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0109] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0110] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A message data consumption method based on a cache mechanism and multi-threading, characterized in that: A storage center for storing data and a data dictionary for filtering data are provided, and the method comprises: Pull message data linearly from the message queue based on a pre-set offset; Splitting the message data into tasks based on a preset number of threads to obtain multiple copies of task data, and distributing the multiple copies of task data by threads; For task data of any thread: Based on the judgment conditions preset in the data dictionary, the input task data is filtered to obtain quasi-linear data or non-quasi-linear data; Wherein, based on the judgment conditions preset in the data dictionary, obtaining quasi-linear data or non-quasi-linear data according to the input task data specifically includes: Extracting a first timestamp from the task data; Obtaining a second timestamp in the data dictionary corresponding to the first timestamp; In response to determining that the first timestamp is later than the second timestamp, filtering the task data through the data dictionary to obtain the quasi-linear data; In response to determining that the first timestamp is earlier than the second timestamp, the task data is filtered through the data dictionary to obtain non-linear data; In response to determining that the quasi-linear data is obtained, performing business logic processing on the quasi-linear data to obtain processed data, and storing the processed data in the storage center; In response to determining that the non-linear data is obtained, storing the non-linear data in the storage center; In response to determining that the total amount of the processed data and the non-linear data in the storage center is equal to the offset, the message data is linearly pulled from the message queue again based on the offset.

2. The method according to claim 1, characterized in that: The obtaining of a second timestamp corresponding to the first timestamp in the data dictionary specifically includes: Determining an index corresponding to the first timestamp in the data dictionary; Based on the index, a second timestamp corresponding to the index in the data dictionary is obtained.

3. The method according to claim 2, characterized in that Determining the index corresponding to the first timestamp in the data dictionary specifically includes: Based on the first timestamp, extracting a linear information field corresponding to the first timestamp from the task data; Determine the data type corresponding to the linear information field; The index corresponding to the data type is obtained from the data dictionary.

4. The method according to claim 1, characterized in that Before storing the processed data in the storage center, the method further includes: The second timestamp in the data dictionary is replaced with the first timestamp.

5. The method according to claim 4, characterized in that Also includes: The replaced second timestamp is stored in the storage center.

6. The method according to claim 5, characterized in that Also includes: In response to determining that the message data has an error, the second timestamp corresponding to the message data is retrieved from the storage center to recover the message data.

7. A message data consumption device based on a cache mechanism and multi-threading, characterized in that: include: The server is configured to: linearly pull message data from the message queue based on a pre-set offset; Splitting the message data into tasks based on a preset number of threads to obtain multiple copies of task data, and distributing the multiple copies of task data by threads; For any thread's task data, it includes: The asynchronous processor is configured to: filter the input task data to obtain quasi-linear data or non-quasi-linear data based on the judgment conditions preset in the data dictionary; Wherein, based on the judgment conditions preset in the data dictionary, obtaining quasi-linear data or non-quasi-linear data according to the input task data specifically includes: Extracting a first timestamp from the task data; Obtaining a second timestamp in the data dictionary corresponding to the first timestamp; In response to determining that the first timestamp is later than the second timestamp, filtering the task data through the data dictionary to obtain the quasi-linear data; In response to determining that the first timestamp is earlier than the second timestamp, the task data is filtered through the data dictionary to obtain non-linear data; In response to determining that the quasi-linear data is obtained, performing business logic processing on the quasi-linear data to obtain processed data, and storing the processed data in a storage center; In response to determining that the non-linear data is obtained, storing the non-linear data in the storage center; The server is further configured to: in response to determining that the total amount of the processed data and the non-linear data in the storage center is equal to the offset, linearly pull data from the message queue again based on the offset.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 6 when executing the program.

9. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 6.

Images