Message scheduling method and device, equipment and medium

Abstract

The application relates to a message scheduling method and device, equipment and medium in the technical field of data processing. The method comprises the following steps: a first consumer reads messages of any merchant in a message queue, determines the synchronization delay of the messages, when the synchronization delay is not high, a thread processes the messages to obtain a processing result and sends the processing result to a downstream; when the synchronization delay is too high and the downstream is in a high-pressure state, a second consumer is created, the first consumer reads messages of important merchants in the message queue, and the second consumer reads messages of ordinary merchants in the message queue; a thread processes the messages in the first consumer to obtain a processing result and sends the processing result to the downstream, the first processing delay of the messages is determined, when the first processing delay is not high, the step is circularly executed; when the first processing delay is too high, a thread processes the messages in the second consumer to obtain a processing result and sends the processing result to the downstream, the second processing delay of the messages is determined, and the previous step is executed. The application can preferentially process messages with higher importance.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a message scheduling method and corresponding apparatus, computer equipment, and computer-readable storage medium. Background Technology

[0002] With the development of technology, real-time streaming data processing technology has become widely used. Kafka is a distributed message publish-subscribe system. Real-time streaming data is generated and published as messages by producers to Kafka topics. Consumers consume messages by subscribing to Kafka topics. Because Kafka is a message queue, it must follow the first-in, first-out (FIFO) principle for reading messages; therefore, it is not possible to have some messages in Kafka read before others.

[0003] Traditional methods involve creating multiple topics, each representing a priority level. Each topic corresponds to a consumer, and a priority consumer group is built, aggregating consumers with different priorities. The number of messages pulled per session by each priority consumer group is set according to business needs, and different quotas are assigned to consumers of different priorities. Consumers consume messages within their respective topics in descending order of priority and according to their quotas. This approach requires creating a large number of topics, consumes significant processing resources, and makes it difficult to switch between priority-based and sequential message reading in Kafka.

[0004] Given the shortcomings of traditional technologies, the applicant has long been engaged in research in related fields and has therefore taken a different approach to solve the industry's problems in the field of data processing technology. Summary of the Invention

[0005] The primary objective of this application is to solve at least one of the above-mentioned problems by providing a message scheduling method and corresponding apparatus, computer equipment, and computer-readable storage medium.

[0006] To achieve the various objectives of this application, the following technical solution is adopted:

[0007] A message scheduling method provided for one of the purposes of this application includes the following steps:

[0008] The first consumer reads messages with any merchant identifier and their generation timestamps from the message queue, determines the synchronization delay of the generation timestamp relative to the current timestamp, and judges whether the synchronization delay exceeds a first preset threshold. If it does not exceed the threshold, the data processing thread processes the message to obtain the processing result and sends the processing result to the downstream database.

[0009] When the first preset threshold is exceeded and the downstream database is under high pressure, a second consumer is created. The first consumer reads messages in the message queue whose merchant identifier belongs to an important merchant, and the second consumer reads messages in the message queue whose merchant identifier belongs to an ordinary merchant.

[0010] The data processing thread reads the message from the first consumer, processes the message to obtain the processing result, sends the processing result to the downstream database, and determines the first processing delay based on the message's generation timestamp and the current timestamp, and determines whether the first processing delay exceeds the second preset threshold.

[0011] When the second preset threshold is exceeded, the previous step is executed repeatedly. When the threshold is not exceeded, the data processing thread reads the message from the second consumer, processes the message to obtain the processing result, sends the processing result to the downstream database, and determines the second processing delay based on the message's generation timestamp and the current timestamp before executing the previous step.

[0012] In a further embodiment, when the second preset threshold is exceeded, the previous step is executed repeatedly. When the threshold is not exceeded, the data processing thread reads the message from the second consumer, processes the message to obtain a processing result, sends the processing result to the downstream database, and determines the second processing delay based on the message's generation timestamp and the current timestamp. After executing the previous step, the following steps are included:

[0013] Using a preset time period as an interval, determine whether the current first processing delay and second processing delay are both lower than the second preset threshold, and whether the downstream database is under high pressure.

[0014] When both the first processing delay and the second processing delay are lower than the second preset threshold, and the downstream database is in a normal state, the second consumer is deleted, and the process jumps to the step of the first consumer reading messages with any merchant identifier in the message queue and generating timestamps to continue execution.

[0015] In a further embodiment, the method further includes the following steps:

[0016] A pressure detector is used to monitor the load status of the downstream database in real time. When the load status is under high pressure, the system queries whether a high pressure identification file exists in the local storage medium. If it does not exist, the system creates a high pressure identification file and stores it in the local storage medium.

[0017] When the load status is normal, query the local storage medium to see if a high-voltage identification file exists. If it exists, delete the high-voltage identification file from the local storage medium.

[0018] In a further embodiment, before the first consumer reads a message with any merchant identifier and its generation timestamp from the message queue, the following steps are included:

[0019] A basic score is determined based on the merchant's basic store data, and a behavioral score is determined based on the merchant's store behavior data.

[0020] The key score of the merchant is obtained by weighted summation of the base score and the behavioral score.

[0021] When a merchant's important rating exceeds the average merchant rating, the merchant's merchant identifier is set to "Important Merchant"; when it is below or equal to the average merchant rating, the merchant's merchant identifier is set to "Regular Merchant".

[0022] In a further embodiment, the message is processed to obtain a processing result, including the following steps:

[0023] Based on the behavior description value corresponding to at least one behavior description key for each product in the message, the behavior description values ​​corresponding to all behavior description keys are weighted and summed to obtain the recommendation rating for the product.

[0024] Recommended products whose recommendation scores meet the preset conditions are selected as the processing results.

[0025] In a further embodiment, the message is processed to obtain a processing result, including the following steps:

[0026] Perform data cleaning operations on the basic product data of multiple products in the message. The data cleaning operations include any one or more of the following operations: cleaning, filtering, deduplication, and handling abnormal data.

[0027] Based on preset business indicators, the basic product data of multiple products after data cleaning are calculated accordingly, and the business indicator values ​​are used as the processing results.

[0028] In a further embodiment, before determining whether the synchronization delay exceeds a first preset threshold, the following steps are included:

[0029] Get the synchronization delay corresponding to each time the first consumer reads a message with any merchant identifier from the message queue within a preset time period;

[0030] The target synchronization delay that meets the preset conditions among the multiple synchronization delays is selected as the first preset threshold.

[0031] On the other hand, a message scheduling device provided to meet one of the purposes of this application includes a single-consumer module, a dual-consumer module, a first message processing module, and a second message processing module. The single-consumer module is used by the first consumer to read messages with arbitrary merchant identifiers and their generation timestamps from the message queue, determine the synchronization delay of the generation timestamp relative to the current timestamp, and determine whether the synchronization delay exceeds a first preset threshold. If it does not exceed the threshold, the data processing thread processes the message to obtain a processing result and sends the processing result to the downstream database. The dual-consumer module is used to create a second consumer when the first preset threshold is exceeded and the downstream database is under high pressure. The first consumer reads messages with merchant identifiers belonging to important merchants from the message queue, and the second consumer processes the messages. The consumer reads messages from the message queue whose merchant identifier belongs to a regular merchant. A first message processing module is used, whereby the data processing thread reads messages from the first consumer, processes the messages to obtain a processing result, sends the processing result to the downstream database, determines a first processing delay based on the message's generation timestamp and current timestamp, and determines whether the first processing delay exceeds a second preset threshold. A second message processing module is used, whereby if the second preset threshold is exceeded, the previous step is executed repeatedly; if the threshold is not exceeded, the data processing thread reads messages from the second consumer, processes the messages to obtain a processing result, sends the processing result to the downstream database, determines a second processing delay based on the message's generation timestamp and current timestamp, and then executes the previous step.

[0032] In a further embodiment, after the second message processing module, there is a: a periodic judgment submodule, used to determine at a preset time period whether the current first processing delay and second processing delay are both lower than a second preset threshold, and whether the downstream database is under high pressure; a single consumer regression submodule, used to delete the second consumer and jump to the step of the first consumer reading messages of any merchant identifier in the message queue and generating timestamps when the first processing delay and second processing delay are both lower than the second preset threshold and the downstream database is in a normal state.

[0033] In a further embodiment, the method further includes: a pressure detection module, used to detect the load status of the downstream database in real time using a pressure detector; when the load status is under high pressure, querying whether a high pressure identification file exists in the local storage medium; if it does not exist, creating a high pressure identification file and storing it in the local storage medium; and an identification deletion module, used to query whether a high pressure identification file exists in the local storage medium when the load status is under normal conditions; if it exists, deleting the high pressure identification file from the local query medium.

[0034] In a further embodiment, before the single-consumer module, there is a sub-item rating determination sub-module, used to determine a basic rating based on the merchant's basic store data and a behavioral rating based on the merchant's store behavior data; an important rating determination sub-module, used to perform a weighted summation of the basic rating and the behavioral rating to obtain the merchant's important rating; and an identifier setting sub-module, used to set the merchant's identifier as an important merchant when the merchant's important rating exceeds the average merchant rating, and set the merchant's identifier as an ordinary merchant when it is lower than or equal to the average merchant rating.

[0035] In a further embodiment, the first message processing module includes: a recommendation rating calculation submodule, used to perform a weighted summation of the behavior description values ​​corresponding to all behavior description keys based on the behavior description values ​​corresponding to at least one behavior description key of each product in the message, to obtain the recommendation rating of the product; and a recommended product filtering submodule, used to filter out recommended products whose recommendation ratings meet preset conditions as the processing result.

[0036] In a further embodiment, the first message processing module includes: a data cleaning submodule, used to perform data cleaning operations on the basic product data of multiple products in the message, the data cleaning operations including any one or more of the following operations: cleaning, filtering, deduplication, and processing abnormal data; and an indicator value calculation submodule, used to perform corresponding calculations on the basic product data of multiple products after the data cleaning operation according to preset business indicators, and obtain business indicator values ​​as processing results.

[0037] In a further embodiment, before the single consumer module, there is a: a synchronization delay acquisition submodule, used to acquire the synchronization delay corresponding to each message read by the first consumer from any merchant identifier in the message queue within a preset time period; and a synchronization delay filtering submodule, used to filter out the target synchronization delay that meets the preset conditions among the multiple synchronization delays as a first preset threshold.

[0038] In another aspect, a computer device provided for one of the purposes of this application includes a central processing unit and a memory, the central processing unit being used to invoke and run a computer program stored in the memory to perform the steps of the message scheduling method described in this application.

[0039] On another aspect, a computer-readable storage medium is provided to suit another purpose of this application, which stores, in the form of computer-readable instructions, a computer program implemented according to the described message scheduling method, which, when invoked by a computer, executes the steps included in the method.

[0040] The technical solution of this application has many advantages, including but not limited to the following aspects:

[0041] This application determines the synchronization delay of a message by having a first consumer read messages from any merchant in the message queue. When the synchronization delay is low, the thread processes the message, obtains the processing result, and sends it downstream. When the synchronization delay is too high, and the downstream is under high pressure, a second consumer is created. The first consumer reads messages from important merchants in the message queue, and the second consumer reads messages from ordinary merchants. The thread processes the messages in the first consumer, obtains the processing result, and sends it downstream, determining the first processing delay of the message. When the first processing delay is low, this step is repeated. When the first processing delay is too high, the thread processes the messages in the second consumer, obtains the processing result, sends it downstream, determines the second processing delay of the message, and then executes the previous step. Therefore, when messages with arbitrary merchant identifiers cannot be processed in a timely manner, the process can switch from processing messages with arbitrary merchant identifiers in the usual way to prioritizing the processing of messages identified as important merchants and delaying the processing of messages identified as ordinary merchants. This ensures that existing processing resources are allocated more to messages of important merchants, allowing these messages to be processed preferentially without requiring additional processing resources throughout the process. Attached Figure Description

[0042] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0043] Figure 1 The network architecture of the e-commerce platform exemplified in this application;

[0044] Figure 2 This is a flowchart illustrating a typical embodiment of the message scheduling method of this application;

[0045] Figure 3 This is a schematic diagram illustrating the process of the first consumer reading messages with arbitrary merchant identifiers when the downstream database returns from a high-pressure state to a normal state in an embodiment of this application.

[0046] Figure 4 This is a schematic diagram illustrating the process of using a pressure detector to detect the load status of a downstream database and configuring a high-pressure identification file in the local storage medium according to the load status in an embodiment of this application.

[0047] Figure 5 This is a flowchart illustrating the process of setting corresponding merchant identifiers based on the merchant's basic store data and store behavior data in an embodiment of this application.

[0048] Figure 6 This is a flowchart illustrating how the recommendation rating of each product in the message is determined in real time in an embodiment of this application, and how recommended products are selected as the processing result based on the recommendation rating.

[0049] Figure 7This is a flowchart illustrating the process of calculating business indicator values ​​as the processing result after real-time data cleaning of basic product data of multiple products in a message, as described in an embodiment of this application.

[0050] Figure 8 This is a schematic diagram of the process for determining the first preset threshold in an embodiment of this application;

[0051] Figure 9 This is a schematic block diagram of the message scheduling device of this application;

[0052] Figure 10 This is a schematic diagram of the structure of a computer device used in this application. Detailed Implementation

[0053] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0054] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0055] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0056] like Figure 1In the network architecture shown, the e-commerce platform 82 is deployed on the Internet to provide corresponding services to its users. Similarly, the devices 80 of the merchant users and the devices 81 of the consumer users of the e-commerce platform 82 are also connected to the Internet to use the services provided by the e-commerce platform.

[0057] An exemplary e-commerce platform 82 provides supply and demand matching of products and / or services to the general public through the Internet infrastructure. In e-commerce platform 82, products and / or services are provided as commodity information. For the sake of simplicity, the concepts of commodity and product are used in this application to refer to the products and / or services in e-commerce platform 82. Specifically, these may be physical products, digital products, tickets, service subscriptions, other offline services, etc.

[0058] In reality, various entities can access e-commerce platform 82 as users and utilize its online services to participate in the business activities facilitated by the platform. These entities can be natural persons, legal persons, or social organizations. Corresponding to the two types of entities in business activities—merchants and consumers—e-commerce platform 82 has two corresponding categories of users: merchant users and consumer users. Entities involved in the product distribution chain in business activities, including manufacturers, sellers, retailers, and logistics providers, can all use online services on e-commerce platform 82 as merchant users. Similarly, consumers in business activities, including actual or potential consumers, can use online services on e-commerce platform 82 as consumer users. In actual business activities, the same entity can operate as both a merchant user and a consumer user; this should be interpreted flexibly.

[0059] The infrastructure used to deploy the e-commerce platform 82 mainly includes the backend architecture and frontend devices. The backend architecture runs various online services through a service cluster, including middleware or frontend services for the platform, services for consumers, and services for merchants, to enrich and improve its service functions. The frontend devices mainly cover the terminal devices used by users as clients to access the e-commerce platform 82, including but not limited to various mobile terminals, personal computers, and point-of-sale devices. For example, merchant users can use their terminal device 80 to enter product information for their online stores or use the interfaces opened by the e-commerce platform to generate their product information; consumer users can use their terminal device 81 to access the webpage of the online store implemented by the e-commerce platform 82, trigger the shopping process by clicking the shopping button provided on the webpage, and call various online services provided by the e-commerce platform 82 during the shopping process to achieve the purpose of placing an order.

[0060] In some embodiments, the e-commerce platform 82 may be implemented via a processing facility including a processor and memory, which stores a set of instructions that, when executed, cause the e-commerce platform 82 to perform the e-commerce and support functions as described in this application. The processing facility may be part of a server, client, network infrastructure, mobile computing platform, cloud computing platform, fixed computing platform, or other computing platform, and may provide electronic components, merchant devices, payment gateways, application developers, marketing channels, transportation providers, customer devices, point-of-sale devices, etc., for the e-commerce platform 82.

[0061] E-commerce platform 82 can provide online services such as cloud computing services, Software as a Service (SaaS), Infrastructure as a Service (IaaS), Platform as a Service (PaaS), Desktop as a Service (DaaS), Hosted Software as a Service, Mobile Backend as a Service (MBaaS), and Information Technology Management as a Service (ITMaaS). In some embodiments, the various functional components of e-commerce platform 82 can be implemented to operate on various platforms and operating systems. For example, for an online store, its administrator user enjoys the same or similar functions regardless of whether it is on iOS, Android, HomonyOS, or a web page.

[0062] E-commerce platform 82 enables merchants to create their own independent websites to run their online stores. It provides merchants with corresponding business management engine instances, allowing them to establish, maintain, and operate one or more online stores across these independent websites. The business management engine instance can be used for content management, task automation, and data management for one or more online stores. It can be configured through interfaces or built-in components to support various specific business processes in the online store, supporting business activities. Independent websites are the infrastructure of e-commerce platform 82, which offers cross-border services. Merchants can maintain their online stores relatively independently and centrally based on these independent websites. Independent websites typically have dedicated domain names and storage space, and different independent websites are relatively independent. E-commerce platform 82 can provide standardized or customized technical support for a large number of independent websites, allowing merchants to customize a business management engine instance that suits their needs and use it to maintain one or more online stores.

[0063] Online stores can be configured and maintained in the backend by merchant users logging into their Business Management Engine instance as administrators. Supported by the various online services provided by the e-commerce platform 82's infrastructure, merchant users can configure various functions within their online stores and view various data as administrators. For example, merchant users can manage various aspects of their online stores, such as viewing recent online store activities, updating the online store's product catalog, managing orders, recent visit activity, and total order activity. Merchant users can also view more detailed information about their business and visitors to their online store by obtaining reports or metrics, such as displaying a sales summary of the merchant's overall business, specific sales and engagement data from promotional sales and marketing channels, etc.

[0064] E-commerce platforms 82 can provide communication facilities and associated merchant interfaces for electronic communication and marketing. For example, they can utilize electronic messaging aggregation facilities to collect and analyze communication interactions between merchants, consumers, merchant devices, customer devices, point-of-sale devices, etc., aggregating and analyzing communications to increase the potential for product sales. For instance, a consumer may have product-related questions, which could lead to a dialogue between the consumer and the merchant (or an automated processor-based agent representing the merchant), where the communication facilities handle the interaction and provide the merchant with analysis on how to increase the probability of a sale.

[0065] In some embodiments, applications suitable for installation on terminal devices can be provided to serve the access needs of different users, enabling various users to access the e-commerce platform 82 by running the application on their terminal devices. Examples include the merchant backend module of online stores within the e-commerce platform 82. During the process of conducting business activities through these functions, the e-commerce platform 82 can implement various functions related to business activities as middleware or online services and expose corresponding interfaces. Then, toolkits corresponding to the interface access functions are embedded into the application to achieve functional expansion and task completion. The business management engine can include a series of basic functions and expose these functions to online services and / or applications via APIs. Online services and applications use the corresponding functions by remotely calling the corresponding APIs.

[0066] With the support of various components of the Business Management Engine instance, the e-commerce platform 82 can provide online shopping functionality, enabling merchants to connect with customers in a flexible and transparent manner. Consumers can select items online, create orders, provide delivery addresses in the orders, and complete payment confirmation. Merchants can then review and fulfill or cancel orders. The review component included with the Business Management Engine instance ensures compliant use of business processes, guaranteeing that orders are suitable for fulfillment before actual execution. Orders may sometimes be fraudulent and require verification (e.g., ID checks). Payment methods that require merchants to wait to ensure receipt of funds can mitigate this risk, and so on. Order risks may arise from fraud detection tools submitted by third parties through order risk APIs, etc. Before fulfillment, merchants may need to obtain or wait to receive payment information to mark the order as paid before preparing to deliver the product. Such situations can all be subject to appropriate review. The review process can be implemented by the fulfillment component. Merchants can leverage fulfillment components to review and adjust operations, and trigger related fulfillment services. These include: manual fulfillment services, used when merchants select and pack products into boxes, purchase shipping labels and enter tracking numbers, or simply mark items as fulfilled; custom fulfillment services, which can define email notifications; API fulfillment services, which can trigger third-party applications to create fulfillment records; legacy fulfillment services, which can trigger custom API calls from the Commerce Management Engine to third parties; and gift card fulfillment services, which can generate and activate gift cards. Merchants can use an order printer application to print shipping documents. The fulfillment process can be executed once items are packed and ready for shipment, tracked, delivered, and verified by the consumer.

[0067] It can be seen that the services provided by e-commerce platforms are based on products, and the corresponding product data is the foundational data of these platforms. Providing product information through this data and mining and utilizing it are fundamental to various technical services. This includes using user transaction data and product data from the e-commerce platform to provide basic services for the operation of a real-time streaming data processing system. Therefore, the real-time streaming data processing system of this application can run on any one or more servers within the e-commerce platform's cluster, enabling the use of various product data provided by the e-commerce platform to achieve various functions.

[0068] The message scheduling method of this application can be programmed into a computer program product and deployed on a client or server for execution. For example, in an exemplary application scenario of this application, it can be deployed on the server of an e-commerce customer service platform. In this way, the method can be executed by human-computer interaction with the process of the computer program product through a graphical user interface by accessing the interface opened after the computer program product is running.

[0069] Please see Figure 2 The message scheduling method of this application, in its typical embodiment, includes the following steps:

[0070] Step S1100: The first consumer reads the message with any merchant identifier and its generation timestamp from the message queue, determines the synchronization delay of the generation timestamp relative to the current timestamp, and determines whether the synchronization delay exceeds a first preset threshold. If it does not exceed the threshold, the data processing thread processes the message to obtain the processing result and sends the processing result to the downstream database.

[0071] A dynamic service configuration center is pre-built using Nacos. In this dynamic service configuration center, the list of important merchants can be dynamically updated. The list of important merchants includes the merchant identifiers of all important merchants on the e-commerce platform. The merchant identifier is used to identify a specific merchant and distinguish it from other merchants. The specific settings can be flexibly modified by those skilled in the art, such as merchant ID.

[0072] When the data source connector is initially started, it first reads the list of important merchants from the dynamic service configuration center. Subsequently, the data source connector establishes a long connection with the dynamic service configuration center to monitor data changes in the list of important merchants in the dynamic service configuration center in real time. When a data change occurs, the list of important merchants in the data source connector can synchronize the data change, so that the list of important merchants in the data source connector always remains the same as the list of important merchants in the dynamic service configuration center.

[0073] The data source connector is built using Flink, a powerful stream processing engine with high throughput, low latency, and exactly-once semantics. Flink also supports integration with various data sources, such as relational databases (MySQL, PostgreSQL, MongoDB), message queues (Kafka, RabbitMQ), and file systems.

[0074] The message queue is built using Kafka, a high-throughput distributed publish-subscribe messaging system. Kafka consists of one or more servers, called brokers. Programs that send messages to brokers are called producers. Programs that subscribe to and consume messages from brokers are called consumers. Each message published to a broker has a category called a topic, and each message belongs to one and only one topic. Each topic has at least one partition, and different partitions within the same topic contain different messages. All brokers, producers, and consumers in the Kafka cluster rely on ZooKeeper to ensure system availability.

[0075] Kafka and Flink integration can perform CDC tasks, which are Change Data Capture (CDC) technologies. CDC tasks synchronize incremental changes from a data source to one or more data sinks. During synchronization, data processing can be performed, such as grouping, joining multiple tables, and numerical calculations. Changes can include INSERT, DELETE, and UPDATE operations. The data sinks are the downstream databases, which can be relational databases (such as MySQL, PostgreSQL, and MongoDB), message queues (such as Kafka and RabbitMQ), Elasticsearch, etc.

[0076] Specifically, regarding the integration of Kafka and Flink, when the data source changes, the corresponding changes are recorded through triggers attached to the data source or binlogs. When the data source is implemented using MySQL, Canal can be used to subscribe to the MySQL binlog. Canal will organize the changed data from the MySQL database into its fixed JSON or Avro format and send it to Kafka as a message. When the data source is implemented using any of the databases such as MySQL, PostgreSQL, or MongoDB, Debezium can be used to capture the changed data, and then organize the changed data into its fixed JSON or Avro format and send it to Kafka as a message.

[0077] Flink serves as a data source connector. To this end, Flink integrates a common Apache Kafka connector, which connects Flink and Kafka, allowing data to be read from or written to a Kafka topic, providing exact-once processing semantics.

[0078] In one embodiment, the data source is implemented using MySQL, with binlog enabled in MySQL to record changes. Canal is used to subscribe to the MySQL binlog, acting as a producer in Kafka. Canal organizes the changed data from the MySQL database into its fixed JSON format and sends it to Kafka as a message. This message is Canal-JSON data, containing a merchant identifier and a generation timestamp. After Kafka receives the message, it sets the received timestamp to the generation timestamp. Whenever a message is stored in Kafka, Flink connected to Kafka starts. The first consumer in Flink acts as a Kafka consumer, subscribing to the corresponding Kafka Topic to read the message with any merchant identifier. At this time, the current timestamp is obtained by calling the System.currentTimeMillis() method. The generation timestamp of the message is subtracted from the current timestamp to calculate the synchronization delay, and it is then determined whether the synchronization delay exceeds a first preset threshold.

[0079] The first preset threshold is used to measure whether the synchronization latency is too long, and it represents an upper limit for the maximum synchronization latency. The specific value can be flexibly set by those skilled in the art based on prior knowledge or experimental data, for example, 2 seconds. The first consumer is built based on the Flink Kafka Consumer and is used to access one or more Kafka topics.

[0080] When the synchronization delay is less than or equal to a first preset threshold, the data processing thread reads the message from the first consumer, uses a JSON deserializer to deserialize the message into RowData format data, and reads the value corresponding to the key "type" from the RowData format data, which is the operation type of the change operation. When the value corresponding to the key "type" is INSERT, the thread iterates through the values ​​corresponding to the key "data" in the RowData format data, which is the data to be inserted, and then adds INSERT as the RowKind, constructs a RowData object, and sends the RowData object as the processing result to the downstream database; when the value corresponding to the key "type" is UPDATE, the thread iterates through the values ​​corresponding to the key "data", which is the data to be inserted. In the RowData format data, the value corresponding to the key "data" is the updated data, and the value corresponding to the key "old" is the data before the update. If the data before the update is empty, it is set as the updated data, and then UPDATE is added as the RowKind to construct a RowData object. This RowData object is sent to the downstream database as the processing result. When the value corresponding to the key "type" is DELETE, the data to be deleted is traversed by looping through the RowData format data with the key "data". Then, a RowKind is added as DELETE to construct a RowData object, and this RowData object is sent to the downstream database as the processing result.

[0081] Step S1200: When the first preset threshold is exceeded and the downstream database is under high pressure, a second consumer is created. The first consumer reads messages in the message queue whose merchant identifier belongs to an important merchant, and the second consumer reads messages in the message queue whose merchant identifier belongs to an ordinary merchant.

[0082] When the synchronization delay exceeds a first preset threshold, in one embodiment, a database connector connected to the downstream database can be activated immediately to monitor the load status of the downstream database and confirm whether the load status is under high pressure. If the load status is under high pressure, it indicates that the downstream database is under high pressure; if the load status is not under high pressure (i.e., it is under normal conditions), it indicates that the downstream database is under normal conditions. In another embodiment, a high-pressure identification file can be queried in the local storage medium in real time. If it exists, it indicates that the downstream database is under high pressure; if it does not exist, it indicates that the downstream database is under normal conditions.

[0083] The high-pressure load state can be manifested as any one or more of the following: storage load, CPU utilization, memory utilization, request latency, and request processing latency of the downstream database exceeding a preset rated value, etc. Those skilled in the art can also flexibly set the specific manifestations of the high-pressure load state according to business needs.

[0084] When the synchronization delay exceeds the first preset threshold and the downstream database is under high pressure, it indicates that the processing results sent to the downstream database are being backed up due to the high pressure on the downstream database. Furthermore, the computing resources of the data connector may be relatively insufficient. In this case, the message processing speed in the data connector is relatively slow compared to normal. Messages with any merchant identifier in the data connector cannot be processed in a timely manner. Therefore, the data connector needs to prioritize processing messages with merchant identifiers belonging to important merchants and process messages with merchant identifiers belonging to ordinary merchants with relative lag.

[0085] To implement the aforementioned priority processing and delayed processing, firstly, a second consumer is created. This second consumer is built based on the Flink Kafka Consumer and is used to access one or more Kafka topics. Then, for messages in the message queue, the first consumer first reads the merchant identifier of the message in the message queue. It then iterates through the important merchant list to determine if the merchant identifier exists. Since the important merchant list contains the merchant identifiers of all important merchants, if the merchant identifier of the message exists in the important merchant list, that is, the merchant identifier belongs to an important merchant, and the first consumer reads the message. If the merchant identifier of the message does not exist in the merchant list, that is, the merchant identifier does not belong to an important merchant but to a regular merchant, the first consumer abandons reading the message, and the second consumer reads the message.

[0086] As can be seen, in the data source connector, messages with merchant identifiers belonging to important merchants are read by the first consumer, while messages with merchant identifiers belonging to ordinary merchants are read by the second consumer, thus achieving message diversion and reading.

[0087] Step S1300: The data processing thread reads the message from the first consumer, processes the message to obtain the processing result, sends the processing result to the downstream database, and determines the first processing delay based on the generation timestamp and the current timestamp of the message, and determines whether the first processing delay exceeds the second preset threshold.

[0088] Furthermore, the data processing thread reads the message from the first consumer, uses a JSON deserializer to deserialize the message into RowData format data, and reads the value corresponding to the key 'type' from the RowData format data, which is the operation type of the change operation. When the value corresponding to the key 'type' is INSERT, the thread iterates through the values ​​corresponding to the key 'data' in the RowData format data, which is the data to be inserted, and then adds INSERT as the RowKind, constructing a RowData object, and sending the RowData object as the processing result to the downstream database; when the value corresponding to the key 'type' is UPDATE ... RowKind, and constructs a RowData object. In the 'ta' format data, the value corresponding to the key 'data' is the updated data, and the value corresponding to the key 'old' is the data before the update. If the data before the update is empty, it is set as the updated data. Then, UPDATE is added as the RowKind, a RowData object is constructed, and this RowData object is sent as the processing result to the downstream database. When the value corresponding to the key 'type' is DELETE, the data to be deleted is iterated through by looping through the RowData format data where the key is 'data', the RowKind is added as DELETE, a RowData object is constructed, and this RowData object is sent as the processing result to the downstream database.

[0089] When sending the processing result corresponding to the message to the downstream database, at the same time, the current timestamp is obtained by calling the System.currentTimeMillis() method, the generation timestamp of the message is subtracted from the current timestamp to obtain the first processing delay, and it is determined whether the first processing delay exceeds the second preset threshold.

[0090] The second preset threshold is used to measure whether the first processing delay and the second processing delay are too long, and it represents an upper limit value for the processing delay. Specifically, it can be flexibly set by those skilled in the art based on prior knowledge or experimental data, for example, 2s.

[0091] Step S1400: When the second preset threshold is exceeded, the previous step is executed repeatedly. When the threshold is not exceeded, the data processing thread reads the message from the second consumer, processes the message to obtain the processing result, sends the processing result to the downstream database, and determines the second processing delay based on the message's generation timestamp and the current timestamp before executing the previous step.

[0092] When the second processing delay exceeds the second preset threshold, the data processing thread continuously processes the messages in the first consumer, obtains the processing results, and sends them to the downstream database, while determining whether the first processing delay of the message exceeds the second preset threshold. When the second processing delay does not exceed the second preset threshold, the data processing thread reads the message from the second consumer, uses a JSON deserializer to deserialize the message into RowData format data, reads the value corresponding to the key 'type' from the RowData format data (i.e., the operation type of the change operation), and when the value corresponding to the key 'type' is INSERT, it iterates through the values ​​corresponding to the key 'data' in the RowData format data (i.e., the data to be inserted), adds INSERT as RowKind, constructs a RowData object, and uses this RowData object as the processing... The processing result is sent to the downstream database. When the value corresponding to the key "type" is UPDATE, the value corresponding to the key "data" in the RowData format data (the updated data) and the value corresponding to the key "old" (the data before the update) are iterated through. If the data before the update is empty, it is set as the updated data. Then, UPDATE is added as the RowKind, a RowData object is constructed, and this RowData object is sent to the downstream database as the processing result. When the value corresponding to the key "type" is DELETE, the value corresponding to the key "data" in the RowData format data (the data to be deleted) is iterated through. Then, a RowKind is added as the DELETE, a RowData object is constructed, and this RowData object is sent to the downstream database as the processing result.

[0093] While sending the processing result corresponding to the message to the downstream database, the current timestamp is obtained by calling the System.currentTimeMillis() method. The message generation timestamp is then subtracted from the current timestamp to obtain the second processing delay. Afterwards, the data processing thread processes the message in the first consumer, obtains the processing result, sends it to the downstream database, and determines whether the first processing delay of the message exceeds the second preset threshold.

[0094] As can be seen from the typical embodiments of this application, the technical solution of this application has many advantages, including but not limited to the following aspects:

[0095] This application determines the synchronization delay of a message by having a first consumer read messages from any merchant in the message queue. When the synchronization delay is low, the thread processes the message, obtains the processing result, and sends it downstream. When the synchronization delay is too high, and the downstream is under high pressure, a second consumer is created. The first consumer reads messages from important merchants in the message queue, and the second consumer reads messages from ordinary merchants. The thread processes the messages in the first consumer, obtains the processing result, and sends it downstream, determining the first processing delay of the message. When the first processing delay is low, this step is repeated. When the first processing delay is too high, the thread processes the messages in the second consumer, obtains the processing result, sends it downstream, determines the second processing delay of the message, and then executes the previous step. Therefore, when messages with arbitrary merchant identifiers cannot be processed in a timely manner, the process can switch from processing messages with arbitrary merchant identifiers in the usual way to prioritizing the processing of messages identified as important merchants and delaying the processing of messages identified as ordinary merchants. This ensures that existing processing resources are allocated more to messages of important merchants, allowing these messages to be processed preferentially without requiring additional processing resources throughout the process.

[0096] Please see Figure 3 In a further embodiment, step S1400, when the second preset threshold is exceeded, the previous step is executed repeatedly; when the threshold is not exceeded, the data processing thread reads the message from the second consumer, processes the message to obtain the processing result, sends the processing result to the downstream database, and determines the second processing delay based on the message's generation timestamp and the current timestamp. After executing the previous step, the following steps are included:

[0097] Step S1500: Using a preset time period as an interval, determine whether the current first processing delay and second processing delay are both lower than the second preset threshold, and whether the downstream database is under high pressure.

[0098] The time period can be set as needed by those skilled in the art, for example, 5 minutes.

[0099] In one embodiment, a database connector connected to a downstream database can be activated immediately to monitor the load status of the downstream database and confirm whether the load status is under high pressure. When the load status is under high pressure, it indicates that the downstream database is under high pressure; when the load status is not under high pressure (i.e., it is in a normal state), it indicates that the downstream database is in a normal state. In another embodiment, a high-pressure identification file can be queried in the local storage medium in real time. If it exists, it indicates that the downstream database is under high pressure; if it does not exist, it indicates that the downstream database is in a normal state.

[0100] Step S1510: When both the first processing delay and the second processing delay are lower than the second preset threshold, and the downstream database is in a normal state, delete the second consumer and jump to the step of the first consumer reading the message of any merchant identifier in the message queue and generating the timestamp to continue execution.

[0101] When both the first processing delay and the second processing delay are lower than the second preset threshold, and the downstream database is in a normal state, it indicates that the processing results sent to the downstream database will not be backed up because the downstream database is in a normal state, and the computing resources of the data connector are normal. At this time, the speed of message processing in the data connector is normal, and messages with any merchant identifier in the data connector can be processed in a timely manner. Therefore, by deleting the second consumer, the data connector is left with only the first consumer. After the first consumer reads the message with any merchant identifier, the data processing thread processes the message and obtains the processing result, which is then sent to the downstream database.

[0102] In this embodiment, by periodically determining whether both the first and second processing delays are below a second preset threshold, and whether the downstream database is under high pressure, if both the first and second processing delays are below the second preset threshold and the downstream database is in a normal state, the second consumer is deleted. This reverts to the original process where the first consumer reads messages with any merchant identifier, the data processing thread processes the message, obtains the processing result, and sends it to the downstream database. Therefore, it ensures that, at appropriate times, the processing priority is given to messages with important merchant identifiers, and messages with ordinary merchant identifiers are processed later, switching to regular processing of messages with any merchant identifier.

[0103] Please see Figure 4 In a further embodiment, the method further includes the following steps:

[0104] Step S2100: Use a pressure detector to detect the load status of the downstream database in real time. When the load status is under high pressure, query whether there is a high pressure identification file in the local storage medium. If not, create a high pressure identification file and store it in the local storage medium.

[0105] A database connector that establishes a long-lived connection to the downstream database serves as a stress detector. This stress detector, deployed on a separate thread or process, monitors the load status of the downstream database to determine if it is under high load. Then, when the load is under high load, it queries the local storage medium to check if a high-load flag file exists; if it does, creation is unnecessary.

[0106] The high-pressure load state can be manifested as any one or more of the following: storage load, CPU utilization, memory utilization, request latency, and request processing latency of the downstream database exceeding a preset rated value, etc. Those skilled in the art can also flexibly set the specific manifestations of the high-pressure load state according to business needs.

[0107] The local storage medium can be memory or a local disk, and those skilled in the art can choose one as needed.

[0108] Step S2200: When the load status is normal, query whether there is a high voltage identification file in the local storage medium. If it exists, delete the high voltage identification file in the local query medium.

[0109] If the load status is not in a high-voltage state, it means that it is in a normal state. At this time, we also check whether there is a high-voltage identification file on the local storage medium. If it does not exist, there is no need to delete it.

[0110] In this embodiment, a pressure detector is used to monitor the load status of the downstream database in real time. When the load status is high, a high-pressure flag file is ensured to exist in the local query medium; when the load status is normal, a high-pressure flag file is ensured to not exist in the local storage medium. Therefore, the presence or absence of a high-pressure flag file in the local query medium can be accurately synchronized in real time to represent the load status of the corresponding downstream database.

[0111] Please see Figure 5 In a further embodiment, before step S1100, where the first consumer reads the message with any merchant identifier and its generation timestamp from the message queue, the following steps are included:

[0112] Step S1000: Determine the basic score based on the merchant's basic store data, and determine the behavioral score based on the merchant's store behavior data.

[0113] The basic store data includes total order volume, total return volume, product description accuracy rating, logistics experience rating, and service experience rating. The store behavior data includes customer service response delay and return response delay.

[0114] The total order volume refers to the total number of orders placed by customers for goods purchased from the merchant's online store. The total order cancellation volume refers to the total number of orders cancelled by customers. The product description accuracy rating is the average of all customer ratings on the accuracy of the product against the merchant's description. The logistics experience rating is the average of all customer ratings on the logistics experience. The service experience rating is the average of all customer ratings on the service experience. The customer service response delay is the average of the time difference between the timestamp of each merchant response and the timestamp of the customer's inquiry. The return response delay is the average of the time difference between the timestamp of each merchant response to a return request and the timestamp of the customer's return request.

[0115] The basic scoring formula is illustrated below:

[0116]

[0117] Where: BaseScore is the base score, order is the total number of orders, chargeback is the total number of refunds, match is the product description accuracy score, logistics is the logistics experience score, service is the service experience score, and matchMax, logisticsMax, and serviceMax are the upper limits for the product description accuracy score, logistics experience score, and service experience score, respectively.

[0118] The behavioral scoring example is shown below:

[0119] BehaviorScore=(replydelay+returndelay) / 2

[0120] Where: BehaviorScore is the behavior score, replydelay is the customer response delay, and returndelay is the return response delay.

[0121] Step S1010: Perform a weighted summation of the basic score and the behavioral score to obtain the merchant's key score;

[0122] It is understood that the basic score and the behavioral score are assigned separate weights, and the sum of the two weights is 1. The weights are used to control the influence of the basic score and the behavioral score on the final calculated important score. Those skilled in the art can set the weights as needed based on the disclosure herein. For example, the weight of the basic score is 0.6 and the weight of the behavioral score is 0.4.

[0123] The important scoring criteria are illustrated with the following example formula:

[0124] ImportantScore=ω1*BaseScore+ω2*BehaviorScore

[0125] Where: ImportantScore is the important score, BaseScore is the basic score, BehaviorScore is the behavior score, ω1 is the weight of the basic score, and ω2 is the weight of the behavior score.

[0126] Step S1020: When the merchant's important rating exceeds the average merchant rating, the merchant's merchant identifier is set as an important merchant; when it is lower than or equal to the average merchant rating, the merchant's merchant identifier is set as an ordinary merchant.

[0127] The average of the important ratings of all merchants on the e-commerce platform is taken as the average rating of the merchants.

[0128] A list of important merchants is created. When a merchant's importance rating exceeds the average merchant rating, it indicates that the merchant's importance exceeds that of more than half of the merchants on the e-commerce platform. Therefore, the merchant is confirmed as an important merchant, and its merchant identifier is stored in the list of important merchants. When a merchant's importance rating is lower than or equal to the average merchant rating, it indicates that the merchant's importance is lower than or equal to that of more than half of the merchants on the e-commerce platform. Therefore, the merchant is confirmed as an ordinary merchant, and its merchant identifier is not stored in the list of important merchants.

[0129] In this embodiment, by determining the basic score and behavioral score of a merchant, and then weighted summing them to obtain the important score, merchants whose important scores exceed the average score of merchants are identified as important merchants, and merchants whose important scores are lower than or equal to the average score of merchants are identified as ordinary merchants. This method can reliably and accurately identify important merchants and ordinary merchants in the e-commerce platform.

[0130] Please see Figure 6 In a further embodiment, step S1300, processing the message to obtain a processing result, includes the following steps:

[0131] Step S1310: Based on the behavior description value corresponding to at least one behavior description key of each product in the message, perform a weighted summation of the behavior description values ​​corresponding to all behavior description keys to obtain the recommendation score of the product.

[0132] The data acquisition system can collect at least one behavioral description key and its corresponding behavioral description value for multiple products in a merchant's online store, which serves as the change data for INSERT operations in the data source. It's easy to understand that the changed data subsequently goes through steps S1100-1200, during which a message corresponding to the changed data is generated. This message is read by either the first or second consumer, depending on the situation. Then, the data processing thread processes the message by reading it from the corresponding consumer. Specifically, a JSON deserializer is used to deserialize the message into RowData format data. The value corresponding to the key "type" in the RowData format data, i.e., the operation type of the change operation, is read. Since the value corresponding to the key "type" is INSERT, the value corresponding to the key "data" in the RowData format data, i.e., the behavioral description value corresponding to at least one behavioral description key for multiple products, is iterated through. For each product, the behavioral description value corresponding to each behavioral description key is multiplied by its weight and then summed to obtain the product's recommendation rating.

[0133] The at least one behavior description key includes any one or more of the following: number of clicks after exposure, number of purchases after click, number of exposures, number of additions to cart after click, number of favorites after click, number of shares after click, and number of returns after purchase. The behavior description value is the specific numerical value of the corresponding behavior description key. The data acquisition system can be further configured as needed by those skilled in the art based on the disclosure herein.

[0134] The sum of the weights of the behavior description values ​​corresponding to each behavior description key is 1. Those skilled in the art can set each weight as needed based on the disclosure here. For example, the behavior description keys include the number of clicks after exposure, the number of purchases after clicking, the number of exposures, the number of times to add to cart after clicking, the number of times to favorites after clicking, the number of times to forward after clicking, and the number of times to return after purchase. The weights of the behavior description values ​​corresponding to these behavior description keys are 0.2, 0.3, 0.1, 0.1, 0.1, 0.1, 0.1, and 0.1, respectively.

[0135] For each product, the behavior description value corresponding to each behavior description key of the product is multiplied by its weight and then summed to obtain the recommendation score for the product.

[0136] Step S1320: Select recommended products whose recommendation scores meet the preset conditions as the processing result.

[0137] Furthermore, based on the recommendation rating of each product, the products are sorted in descending order of recommendation rating. The top N products are selected, as they have higher recommendation ratings than the others, meaning they have higher recommendation value. Therefore, these N products are designated as recommended products. A RowData object is constructed by combining these recommended products with a RowKind added via INSERT. This RowData object is then sent as the processing result to the downstream database. N is less than the number of products in the message and can be set as needed by those skilled in the art.

[0138] In this embodiment, by calculating a corresponding recommendation score for each product in the message, and then selecting recommended products as the processing result, it is possible to accurately determine suitable recommended products from multiple products in real-time data in a timely manner.

[0139] Please see Figure 7 In a further embodiment, step S1300, processing the message to obtain a processing result, includes the following steps:

[0140] Step S1301: Perform data cleaning operation on the basic product data of multiple products in the message. The data cleaning operation includes any one or more of the following operations: cleaning, filtering, deduplication, and processing abnormal data.

[0141] The data acquisition system can collect basic product data from multiple products in a merchant's online store, which will then be used as the data source for INSERT operations. It's easy to understand that the changed data will subsequently go through steps S1100-1200, during which a message corresponding to the changed data will be generated. This message will be read by either the first or second consumer, depending on the situation. Then, the data processing thread will process the message by reading it from the corresponding consumer. Specifically, a JSON deserializer will be used to deserialize the message into RowData format data. The value corresponding to the key "type" in the RowData format data, which is the operation type of the change operation, will be read. Since the value corresponding to the key "type" is INSERT, the basic product data corresponding to the key "data" in the RowData format data, which is the basic product data of multiple products, will be iterated through to perform data cleaning operations on this basic product data.

[0142] The basic product data includes at least one attribute description key and its attribute description value for the corresponding product. The at least one attribute description key includes any one or more of the following: product category, product tag, generation timestamp, product title, detailed parameters, sales volume, and price. The attribute description value is the specific numerical value or text corresponding to the attribute description key. The data acquisition system can be further configured as needed by those skilled in the art based on the disclosure herein.

[0143] The data cleaning operations described herein can be configured as needed by those skilled in the art. Examples include: Cleaning: Cleaning invalid or redundant data in the dataset, such as deleting missing values ​​or incomplete records. Filtering: Selecting basic product data that meets specific conditions, such as retaining only basic product data within a certain time range or basic product data of a specific category. Deduplication: Removing duplicate basic product data to ensure that each product's basic product data appears only once. Handling abnormal data: Processing abnormal or unreasonable basic product data, such as correcting incorrect prices or sales volumes.

[0144] Step S1302: Calculate the basic product data of multiple products after data cleaning based on preset business indicators, and obtain the business indicator values ​​as the processing results.

[0145] The preset business indicators can be any one or more of the following: total sales, average price, etc., and those skilled in the art can set them as needed.

[0146] Furthermore, the cleaned data is calculated based on the preset business indicators, such as summation, averaging, and sorting, to obtain the corresponding business indicator values. For example, to calculate the total sales amount, the sales volume of each product can be multiplied by its price, and then the summation can be applied to all products; to calculate the average price, the prices of all products can be averaged.

[0147] All the obtained business metric values ​​and the RowKind added by INSERT are used to construct a RowData object, which is then sent as the processing result to the downstream database.

[0148] In this embodiment, by performing data cleaning operations on the basic product data of multiple products in the message, and then calculating the corresponding business indicator values, the business indicator values ​​can be determined in a timely and accurate manner.

[0149] Please see Figure 8 In a further embodiment, before step S1100, determining whether the synchronization delay exceeds a first preset threshold, the following steps are included:

[0150] Step S1101: Obtain the synchronization delay corresponding to each message read by the first consumer from any merchant identifier in the message queue within a preset time period;

[0151] To ensure reliability, the preset duration should be set to more than three days. The specific duration can be set as needed by those skilled in the art based on the disclosure herein, with a recommended preset duration of one week.

[0152] It is understandable that when messages from any merchant in the data connector can be processed in a timely manner, the first consumer reads messages from any merchant in the message queue, obtains the current timestamp by calling the System.currentTimeMillis() method, and subtracts the message's generation timestamp from the current timestamp to obtain the synchronization delay.

[0153] Step S1102: Select the target synchronization delay that meets the preset conditions from the multiple synchronization delays as the first preset threshold.

[0154] All synchronization delays are collected and sorted in ascending order to obtain a synchronization delay sequence. The top N synchronization delays are deleted to exclude synchronization delays caused by extreme cases. Then, the first synchronization delay in the sequence is selected as the first preset threshold. The number N can be set as needed by those skilled in the art.

[0155] This embodiment discloses the process of determining the first preset threshold, which enables the scientific and effective setting of a suitable first preset threshold.

[0156] Please see Figure 9This message scheduling device, provided to meet one of the purposes of this application, is a functional embodiment of the message scheduling method of this application. On another aspect, this message scheduling device, provided to meet one of the purposes of this application, includes a single-consumer module 1100, a dual-consumer module 1200, a first message processing module 1300, and a second message processing module 1400. The single-consumer module 1100 is used by the first consumer to read messages with arbitrary merchant identifiers and their generation timestamps from the message queue, determine the synchronization delay of the generation timestamp relative to the current timestamp, and determine whether the synchronization delay exceeds a first preset threshold. When it does not exceed the threshold, the data processing thread processes the message to obtain a processing result and sends the processing result to the downstream database. The dual-consumer module 1200 is used to create a second consumer when the first preset threshold is exceeded and the downstream database is under high pressure. A consumer reads messages from the message queue whose merchant identifier belongs to an important merchant, and a second consumer reads messages from the message queue whose merchant identifier belongs to an ordinary merchant. A first message processing module 1300 is used to have the data processing thread read messages from the first consumer, process the messages to obtain a processing result, send the processing result to the downstream database, and determine a first processing delay based on the message's generation timestamp and current timestamp, and determine whether the first processing delay exceeds a second preset threshold. A second message processing module 1400 is used to repeatedly execute the previous step when the second preset threshold is exceeded; when the threshold is not exceeded, the data processing thread reads messages from the second consumer, processes the messages to obtain a processing result, sends the processing result to the downstream database, and after determining a second processing delay based on the message's generation timestamp and current timestamp, executes the previous step.

[0157] In a further embodiment, after the second message processing module 1400, there is a: a periodic judgment submodule, used to determine whether the current first processing delay and second processing delay are both lower than a second preset threshold, and whether the downstream database is under high pressure, at a preset time period; and a single consumer regression submodule, used to delete the second consumer and jump to the step of the first consumer reading messages of any merchant identifier in the message queue and generating timestamps when the first processing delay and second processing delay are both lower than the second preset threshold and the downstream database is in a normal state.

[0158] In a further embodiment, the method further includes: a pressure detection module, used to detect the load status of the downstream database in real time using a pressure detector; when the load status is under high pressure, querying whether a high pressure identification file exists in the local storage medium; if it does not exist, creating a high pressure identification file and storing it in the local storage medium; and an identification deletion module, used to query whether a high pressure identification file exists in the local storage medium when the load status is under normal conditions; if it exists, deleting the high pressure identification file from the local query medium.

[0159] In a further embodiment, before the single consumer module 1100, there are: a sub-item rating determination sub-module, used to determine a basic rating based on the merchant's basic store data and a behavioral rating based on the merchant's store behavior data; an important rating determination sub-module, used to perform a weighted summation of the basic rating and the behavioral rating to obtain the merchant's important rating; and an identifier setting sub-module, used to set the merchant's identifier as an important merchant when the merchant's important rating exceeds the average merchant rating, and set the merchant's identifier as an ordinary merchant when it is lower than or equal to the average merchant rating.

[0160] In a further embodiment, the first message processing module 1300 includes: a recommendation rating calculation submodule, used to perform a weighted summation of the behavior description values ​​corresponding to all behavior description keys based on the behavior description values ​​corresponding to at least one behavior description key of each product in the message, to obtain the recommendation rating of the product; and a recommended product filtering submodule, used to filter out recommended products whose recommendation ratings meet preset conditions as processing results.

[0161] In a further embodiment, the first message processing module 1300 includes: a data cleaning submodule, used to perform data cleaning operations on the basic product data of multiple products in the message, the data cleaning operations including any one or more of the following operations: cleaning, filtering, deduplication, and processing abnormal data; and an indicator value calculation submodule, used to perform corresponding calculations on the basic product data of multiple products after the data cleaning operation according to preset business indicators, and obtain business indicator values ​​as processing results.

[0162] In a further embodiment, before the single consumer module 1100, there are: a synchronization delay acquisition submodule, used to acquire the synchronization delay generated each time the first consumer reads a message with any merchant identifier in the message queue within a preset time period; and a synchronization delay filtering submodule, used to filter out the target synchronization delay that meets the preset conditions among the multiple synchronization delays as a first preset threshold.

[0163] To address the aforementioned technical problems, embodiments of this application also provide computer equipment. For example... Figure 10The diagram shows the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable storage medium stores an operating system, a database, and computer-readable instructions. The database may store control information sequences. When the computer-readable instructions are executed by the processor, they enable the processor to implement a message scheduling method. The processor of the computer device provides computing and control capabilities, supporting the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When these computer-readable instructions are executed by the processor, they enable the processor to execute the message scheduling method of this application. The network interface of the computer device is used for communication with a terminal. Those skilled in the art will understand that… Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0164] In this embodiment, the processor is used to execute... Figure 9 The system defines the specific functions of each module and its submodules. The memory stores the program code and various data required to execute these modules or submodules. The network interface is used for data transmission between the user terminal and the server. In this embodiment, the memory stores the program code and data required to execute all modules / submodules in the message scheduling device of this application. The server can call the server's program code and data to execute the functions of all submodules.

[0165] This application also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the message scheduling method of any embodiment of this application.

[0166] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0167] In summary, this application, on the one hand, can switch from processing messages with any merchant identifier in a timely manner to prioritizing messages from important merchants and delaying the processing of messages from ordinary merchants when it is not possible to process such messages in a timely manner. This ensures that more processing resources are allocated to messages from important merchants, allowing them to be processed first. Then, when messages with any merchant identifier can be processed in a timely manner, it can switch back without requiring additional processing resources. On the other hand, the ability to dynamically synchronize the setting of merchant identifiers for messages provides a certain degree of flexibility.

[0168] Those skilled in the art will understand that the steps, measures, and solutions in the various operations, methods, and processes discussed in this application can be alternated, modified, combined, or deleted. Furthermore, other steps, measures, and solutions in the various operations, methods, and processes discussed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted. Furthermore, steps, measures, and solutions in the prior art that are similar to those in the open-source operations, methods, and processes of this application can also be alternated, modified, rearranged, decomposed, combined, or deleted.

[0169] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A message scheduling method, characterized in that, Includes the following steps: The first consumer reads messages with any merchant identifier and their generation timestamps from the message queue, determines the synchronization delay of the generation timestamp relative to the current timestamp, and judges whether the synchronization delay exceeds a first preset threshold. If it does not exceed the threshold, the data processing thread processes the message to obtain the processing result and sends the processing result to the downstream database. When the first preset threshold is exceeded and the downstream database is under high pressure, a second consumer is created. The first consumer reads messages in the message queue whose merchant identifier belongs to an important merchant, and the second consumer reads messages in the message queue whose merchant identifier belongs to an ordinary merchant. The data processing thread reads messages from the first consumer, processes the messages to obtain processing results, sends the processing results to the downstream database, and determines a first processing delay based on the message's generation timestamp and current timestamp. It then determines whether the first processing delay exceeds a second preset threshold. This process includes: weighted summing of the behavior description values ​​corresponding to all behavior description keys for each product in the message, based on the behavior description value corresponding to at least one behavior description key in the message, to obtain a recommendation rating for the product; and selecting recommended products whose recommendation ratings meet preset conditions as the processing result. When the second preset threshold is exceeded, the previous step is executed repeatedly. When the threshold is not exceeded, the data processing thread reads the message from the second consumer, processes the message to obtain the processing result, sends the processing result to the downstream database, and determines the second processing delay based on the message's generation timestamp and the current timestamp before executing the previous step. This includes: performing data cleaning operations on the basic product data of multiple products in the message, the data cleaning operations including any one or more of the following operations: cleaning, filtering, deduplication, and processing abnormal data; and performing corresponding calculations on the basic product data of multiple products after the data cleaning operation according to preset business indicators to obtain business indicator values ​​as the processing result.

2. The message scheduling method according to claim 1, characterized in that, When the second preset threshold is exceeded, the previous step is executed repeatedly. When the threshold is not exceeded, the data processing thread reads the message from the second consumer, processes the message to obtain the processing result, sends the processing result to the downstream database, and determines the second processing delay based on the message's generation timestamp and the current timestamp. After executing the previous step, the following steps are included: Using a preset time period as an interval, determine whether the current first processing delay and second processing delay are both lower than the second preset threshold, and whether the downstream database is under high pressure. When both the first processing delay and the second processing delay are lower than the second preset threshold, and the downstream database is in a normal state, the second consumer is deleted, and the process jumps to the step of the first consumer reading messages with any merchant identifier in the message queue and generating timestamps to continue execution.

3. The message scheduling method according to claim 1, characterized in that, This method also includes the following steps: A pressure detector is used to monitor the load status of the downstream database in real time. When the load status is under high pressure, the system queries whether a high pressure identification file exists in the local storage medium. If it does not exist, the system creates a high pressure identification file and stores it in the local storage medium. When the load status is normal, check if a high-voltage identification file exists in the local storage medium. If it exists, delete the high-voltage identification file from the local storage medium.

4. The message scheduling method according to claim 1, characterized in that, Before the first consumer reads a message with any merchant identifier and its generation timestamp from the message queue, the following steps are included: A basic score is determined based on the merchant's basic store data, and a behavioral score is determined based on the merchant's store behavior data. The key score of the merchant is obtained by weighted summation of the base score and the behavioral score. When a merchant's important rating exceeds the average merchant rating, the merchant's merchant identifier is set to "Important Merchant"; when it is below or equal to the average merchant rating, the merchant's merchant identifier is set to "Regular Merchant".

5. The message scheduling method according to claim 1, characterized in that, Before determining whether the synchronization delay exceeds a first preset threshold, the following steps are included: Get the synchronization delay corresponding to each time the first consumer reads a message with any merchant identifier from the message queue within a preset time period; Select the target synchronization delay that meets the preset conditions from multiple synchronization delays as the first preset threshold.

6. A message scheduling device, characterized in that, include: The single consumer module is used by the first consumer to read messages with any merchant identifier and their generation timestamps from the message queue, determine the synchronization delay of the generation timestamp relative to the current timestamp, and determine whether the synchronization delay exceeds a first preset threshold. If it does not exceed the threshold, the data processing thread processes the message to obtain the processing result and sends the processing result to the downstream database. The dual-consumer module is used to create a second consumer when the first preset threshold is exceeded and the downstream database is under high pressure. The first consumer reads messages in the message queue whose merchant identifier belongs to an important merchant, and the second consumer reads messages in the message queue whose merchant identifier belongs to an ordinary merchant. The first message processing module is used by the data processing thread to read messages from the first consumer, process the messages to obtain processing results, send the processing results to the downstream database, determine a first processing delay based on the message's generation timestamp and current timestamp, and determine whether the first processing delay exceeds a second preset threshold. This includes: weighting and summing the behavior description values ​​corresponding to all behavior description keys based on the behavior description values ​​corresponding to at least one behavior description key for each product in the message to obtain a recommendation rating for the product; and selecting recommended products whose recommendation ratings meet preset conditions as the processing result. The second message processing module is used to repeatedly execute the previous step when the second preset threshold is exceeded. When the threshold is not exceeded, the data processing thread reads the message from the second consumer, processes the message to obtain the processing result, sends the processing result to the downstream database, and determines the second processing delay based on the message's generation timestamp and the current timestamp before executing the previous step. This includes: performing data cleaning operations on the basic product data of multiple products in the message, the data cleaning operations including any one or more of the following operations: cleaning, filtering, deduplication, and handling abnormal data; and performing corresponding calculations on the basic product data of multiple products after the data cleaning operation according to preset business indicators to obtain business indicator values ​​as the processing result.

7. The message scheduling device according to claim 6, characterized in that, Following the second message processing module, the system includes: a periodic judgment submodule, used to determine whether the current first processing delay and second processing delay are both lower than a second preset threshold, and whether the downstream database is under high pressure, at preset time intervals; and a single consumer regression submodule, used to delete the second consumer and jump to the step of the first consumer reading messages with any merchant identifier in the message queue and generating timestamps when the first processing delay and second processing delay are both lower than the second preset threshold and the downstream database is in a normal state.

8. The message scheduling device according to claim 6, characterized in that, Prior to the single-consumer module, the system includes: a sub-item rating determination sub-module, used to determine a basic rating based on the merchant's basic store data and a behavioral rating based on the merchant's store behavior data; an important rating determination sub-module, used to perform a weighted summation of the basic rating and the behavioral rating to obtain the merchant's important rating; and an identifier setting sub-module, used to set the merchant's identifier as an important merchant when the merchant's important rating exceeds the average merchant rating, and to set the merchant's identifier as an ordinary merchant when it is lower than or equal to the average merchant rating.

9. A computer device comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 5, which, when invoked by a computer, executes the steps included in the corresponding method.

Images