Customer information search method, device, equipment and product based on search engine

Abstract

The invention discloses a customer information searching method and device based on a search engine, equipment and a product. The method comprises the steps of capturing a change event of first-class customer data in real time by analyzing a binary log of a basic database, and sending the change event to message-oriented middleware, the first-class customer data comprising basic attribute information of a customer, and the basic attribute information of the customer; the basic database is used for storing customer information generated by upstream business activities in the form of a plurality of data tables; sending a service event message generated by the upstream service activity to the message-oriented middleware, wherein the service event message comprises a service event type, an identifier of a customer participating in the service event, an identifier of a person and the occurrence time of the service event; and obtaining a second type of client data based on the client information in the data warehouse, and sending the second type of client data to the message middleware. By utilizing the embodiment of the invention, the data query efficiency can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, and also relates to data synchronization and data search technology, in particular to a customer information search method and device based on a search engine, equipment, program product and storage medium. BACKGROUND

[0002] With the deepening of enterprise digital transformation, customer information has become the core asset of enterprises. Effectively managing, querying and analyzing customer information is crucial for customer relationship maintenance, precision marketing and business decision-making. At present, enterprises mainly rely on traditional relational databases (such as MySQL, Oracle) for customer information query. Under this architecture, customer data is often scattered in multiple different data tables according to business modules, such as customer basic information table, sales contact record table, order history table, user behavior log table, etc.

[0003] When business personnel need to perform complex multi-dimensional combined queries (for example, find customers in a specific city, who have not been contacted recently, have reached a certain standard of historical consumption, and are active recently), the system needs to perform multi-table association query and complex condition filtering in the database, resulting in prolonged response time, which cannot meet the demand of business personnel for real-time response, and reduces user experience and business efficiency.

[0004] In addition, in the field of data search, in order to solve the query performance bottleneck, a search engine (such as Elasticsearch, Solr) is often used to build a unified customer information retrieval center. By building data as a search-oriented wide table index, millisecond-level retrieval is achieved using inverted index. However, since the customer data sources are scattered in various independent business systems (such as CRM, ERP, order system, user behavior analysis platform), the existing method of synchronizing customer data to search data often has high system iteration and maintenance cost, data delay of up to several hours or even a day, resulting in inaccurate query results and data lag problems. SUMMARY

[0005] Therefore, the embodiments of the present application provide a customer information search method and device based on a search engine, electronic equipment, computer readable storage medium and computer program product, which solve at least one technical problem.

[0006] The embodiment of the present application provides a customer information search method based on a search engine, comprising the following steps: capturing a change event of first customer data in real time by analyzing a binary log of a basic database, and sending the change event to a message middleware, wherein the first customer data comprises basic attribute information of a customer, and the basic database is used for storing customer information generated by upstream business activities in the form of a plurality of data tables; sending a business event message generated by the upstream business activities to the message middleware, wherein the business event message comprises a business event type, a customer identifier participating in the business event, a personnel identifier, and a business event occurrence time; obtaining second customer data based on customer information in a data warehouse, and sending the second customer data to the message middleware; wherein the second customer data is a customer statistical index obtained through offline batch calculation, and comprises at least one of whether customer historical income meets a preset condition and an order behavior index of the customer in a preset historical interval; listening to the message middleware, if there is a change message related to the first customer data or the second customer data in the message middleware, updating an index field of a corresponding customer document in a target search engine based on the change message, so as to realize real-time synchronization of the first customer data or the second customer data; if there is the business event message in the message middleware, generating third customer data through real-time stream calculation based on the business event message, and updating the index field of the corresponding customer document in the target search engine; wherein the third customer data is a customer behavior index that needs to be calculated in real time, and comprises at least one of whether the customer is contacted by a salesperson in a preset time and an activity index of the customer in a preset period; and in response to a customer information query request initiated by a user, searching the index in the target search engine and returning a search result, wherein the query request comprises at least one query condition.

[0007] Optionally, according to the method of the embodiment of the present application, the real-time calculation of the third customer data based on the business event message comprises: updating corresponding customer state data based on the customer identifier in the business event message in response to the reception of the business event message; and calculating the third customer data in real time according to the updated state data and storing the third customer data in the form of a Boolean field in the index.

[0008] Optionally, according to the method of the embodiment of the present application, the method further comprises: allocating a unique tracking identifier to at least part of the synchronized data; periodically sampling and comparing the consistency of corresponding data in the basic database or the data warehouse and the target search engine; and when inconsistency is identified, automatically triggering a resynchronization or repair task of the corresponding data based on the unique tracking identifier.

[0009] Optionally, according to the method of the embodiment of the present application, the target search engine is an Elasticsearch distributed search and analysis engine.

[0010] Optionally, in the method according to the embodiments of this application, the step of updating the obtained second-type customer data or third-type customer data to the target search library adopts batch submission optimization: multiple single data write operations for the target search library are merged into a batch request; the batch request is submitted to the target search engine for execution at once.

[0011] Optionally, according to the method of the embodiments of this application, in some embodiments of this application, before updating the third type of customer data to the target search engine, the method may further include: converting the raw indicator values ​​obtained by streaming computation into one or more corresponding Boolean or enumerated pre-calculated field values ​​based on one or more predefined query condition templates, and storing these pre-calculated fields as independent, indexable fields.

[0012] This application proposes a customer information search device based on a search engine, comprising: a first type of search data acquisition module, used to acquire first type of search data from a basic database, the first type of search data including basic attribute information of customers, wherein the basic database is used to store customer information generated by upstream business activities in the form of multiple data tables; a second type of search data generation module, used to acquire business event messages generated by upstream business activities, and generate second type of search data based on the business event messages through real-time stream computing; wherein the business event messages include business event type, customer identifiers and personnel identifiers participating in the business event, and the time of occurrence of the business event; the second type of customer data consists of customer behavior indicators that need to be calculated in real time, including at least one of the following: whether the customer was contacted by sales within a preset time period, and the customer's activity level within a preset period; and a third type of search data acquisition module. The module retrieves third-type search data based on customer information in a data warehouse. This third-type customer data consists of customer statistical indicators obtained through offline batch calculations, including at least one of the following: whether the customer's historical revenue meets preset conditions, and customer order behavior indicators within a preset historical period. The data warehouse is used to offline batch store customer information from the basic database according to a preset period. The update module updates the retrieved search data to the search engine. First-type search data is updated to the search engine by adding an index, while second-type and third-type search data are updated to the target search database and then to the search engine by updating index fields. The search module retrieves search results from the index in the target search engine based on user-inputted or selected keywords.

[0013] This application provides an electronic device, which includes a processor and a memory storing computer program instructions; the electronic device executes the computer program instructions to implement the method described above.

[0014] This application provides a computer program product, which includes computer program instructions that, when executed, implement the method described above.

[0015] This application provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the method described above.

[0016] This application embodiment establishes three different data synchronization mechanisms based on the data types of the search engine. This enables efficient and reliable, low-latency synchronization of customer basic information, real-time behavioral indicators, and historical statistical indicators to the search engine, ensuring data accuracy and timeliness even after changes occur in any business system. Secondly, the search engine directly stores the calculation results of customer behavioral indicators and historical statistical indicators, improving retrieval efficiency. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings of the embodiments of this application will be briefly described below.

[0018] Figure 1 This is a schematic diagram of the system architecture of an embodiment of this application.

[0019] Figure 2 This is a flowchart of a customer information search method based on a search engine, according to an embodiment of this application.

[0020] Figure 3 This is a diagram illustrating the fields included in a basic database according to an embodiment of this application.

[0021] Figure 4 This is a schematic diagram summarizing the overall processing procedure of the customer information search method based on a search engine according to an embodiment of this application.

[0022] Figure 5 This is a schematic diagram of the data synchronization process of the customer information search method based on a search engine according to an embodiment of this application.

[0023] Figure 6 This is a schematic structural block diagram of a customer information search device based on a search engine, according to an embodiment of this application.

[0024] Figure 7 This is a schematic diagram of an electronic device used to implement the customer information search method based on a search engine according to the embodiments of this application. Detailed Implementation

[0025] The principles and spirit of this application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided to make the principles and spirit of this application clearer and more thorough. The exemplary embodiments provided herein are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application.

[0026] Embodiments of this application relate to terminal devices and / or servers. Implementations of this application can be a system, terminal, device, method, computer-readable storage medium, or computer program product, and can be specifically implemented as entirely hardware, entirely software, or a combination of hardware and software. Figure 1 This diagram illustrates a system architecture according to an embodiment of the present application, including a terminal device 102 and a server 104. The terminal device 102 may include at least one of the following: a smartphone, tablet computer, laptop computer, desktop computer, smart TV, various wearable devices, augmented reality (AR) devices, virtual reality (VR) devices, etc. A client application, such as an app, mini-program, or browser-based client, can be installed on the terminal device 102. Users can input commands through the client, and the terminal device 102 can send request information containing the commands to the server 104. Upon receiving the request information, the server 104 performs corresponding processing and returns the processing result information to the terminal device 102. The server 104 may be a local server or a cloud server, and may be a single server or a server cluster, etc.

[0027] In this document, the terms "first," "second," "third," etc., are used only to distinguish one entity (or operation) from another in textual description, and do not require or imply any sequential order between these entities (or operations).

[0028] Figure 2 The diagram illustrates a flowchart of a customer information retrieval method based on a search engine, according to an embodiment of this application. The method includes the following steps: S101: By parsing the binary logs of the basic database, change events of the first type of customer data are captured in real time, and the change events are sent to the message middleware. The first type of customer data includes the basic attribute information of customers. The basic database is used to store customer information generated by upstream business activities in the form of multiple data tables. S102: Send the business event messages generated by upstream business activities to the message middleware. The business event messages include the business event type, the customer identifier and personnel identifier of the participating business event, and the time when the business event occurred. S103: Obtain the second type of customer data based on customer information in the data warehouse, and send the second type of customer data to the message middleware; wherein, the second type of customer data is customer statistical indicators obtained through offline batch calculation, including at least one of the following: whether the customer's historical revenue meets the preset conditions, and the customer's order behavior indicators within the preset historical period; S104: In the message listening middleware, if there is a change message related to the first type of customer data or the second type of customer data in the message middleware, then update the index field of the corresponding customer document in the target search engine based on the change message to achieve real-time synchronization of the first type of customer data or the second type of customer data. S105: If there are business event messages in the message middleware, generate third-type customer data based on the business event messages using real-time stream computing, and update the index field of the corresponding customer document in the target search engine; wherein, the third-type customer data are customer behavior indicators that need to be calculated in real time, including at least one of the following: whether the customer was contacted by sales within a preset time period, and the customer's activity level within a preset period. S106: In response to a customer information query request initiated by a user, retrieve the index in the target search engine and return the search results, wherein the query request contains at least one query condition.

[0029] In this embodiment, the target search engine is a distributed, highly scalable, and real-time search and data analysis engine, such as Elasticsearch, with the following two functions: First, it has an inverted index capability, which can build an efficient index structure for fields such as text, numbers, dates, and even boolean values, enabling multi-condition combined filtering and full-text search to achieve millisecond-level response, which directly meets the stringent requirements of the solution for query performance; Second, it has a distributed architecture, which allows index data to be sharded and stored and processed on multiple nodes, providing near-linear horizontal scalability.

[0030] This application categorizes the data in the search engine as follows: Basic attribute information (such as city and status) directly from the business database, frequently changing, and requiring strong consistency is defined as the first type of customer data; behavioral indicators derived from business events and requiring lightweight real-time computation (such as "whether the user has been contacted in the last 365 days") are defined as the third type of customer data; and statistical indicators involving large historical datasets, computationally complex, but with high tolerance for timeliness (such as "whether the user's income in the past 5 years has exceeded 50,000") are defined as the second type of customer data. This classification method is based on a comprehensive judgment of data source, computational complexity, and business tolerance for delays.

[0031] Based on the above classification, this application presents three parallel data processing pipelines. For the first type of customer data, a precise capture mechanism based on database transaction logs is adopted. Specifically, by parsing the binary log (BinLog, a log file that records all data change operations in the database) of the customer database in real time, the addition, deletion, and modification events of each row of data can be captured non-intrusively, just like listening to the master-slave replication stream of the database itself. These change events are encapsulated into standard messages and sent to a central message middleware (such as Kafka). An independent synchronization service listens to messages related to the basic data in this middleware and calls the search engine's API accordingly to update the index fields in the corresponding customer documents. This path ensures that any changes to core business data can be reliably reflected on the search end within a very short latency (usually down to the second or even millisecond level).

[0032] For the third type of customer data (real-time calculated metrics), the processing logic shifts to event-driven and stream computing. Upstream business applications proactively send structured business event messages when key actions occur (e.g., sales submit visit records), containing key metadata such as event type, involved customer and staff identifiers, and occurrence time. These messages are also sent to a unified message middleware. A dedicated real-time stream computing service (utilizing frameworks like Flink or Spark Streaming) continuously consumes these event streams, maintaining or quickly querying the latest customer status in memory, and performing real-time calculations based on predefined business rules (e.g., determining if the difference between the current time and the most recent contact timestamp exceeds 365 days). The calculated result (e.g., a boolean label indicating "no contact within 365 days") is synchronized to the corresponding customer index in the search engine by the search engine's update service. The key to this process is that the complex conditional judgments required during querying are performed earlier in the data writing stage, and the results are materialized into index fields that can be directly used for fast filtering.

[0033] For the second type of customer data (offline statistical indicators), a periodic batch calculation and synchronization strategy was adopted. Raw business data stored in the basic database is periodically (every evening) synchronized to the data warehouse using tools such as ETL. Within the data warehouse, leveraging its powerful batch processing capabilities, calculation tasks are run according to a preset cycle (e.g., daily) to aggregate and analyze historical orders, behavior logs, etc., producing the required statistical indicators. After the calculation results are generated, they are injected into a unified message middleware queue in the form of messages through offline synchronization tasks. Then, according to certain business rules, the search engine's update service is called to synchronize the data, ultimately updating these indicator data in the search engine's customer documents. This path efficiently and economically solves the challenge of in-depth analysis of massive historical data with controllable latency.

[0034] All data processed through the three paths described above is ultimately aggregated and stored uniformly in the target search engine, forming a "wide table"-style document index with the customer identifier as the primary key, incorporating their basic attributes, real-time behavioral tags, and historical statistical tags. When a user initiates a complex query request containing multi-dimensional combined conditions, the query engine no longer needs to return to the underlying database to perform complex multi-table joins and full table scans, nor does it need to calculate metrics during the query. Instead, it directly sends a request to the search engine. The search engine utilizes its efficient retrieval mechanisms, such as inverted indexes, to quickly combine, filter, and sort the pre-calculated and indexed fields on a single customer index, thereby returning accurate query results within milliseconds.

[0035] In addition, in this application, the message middleware unifies the data access and distribution methods from different sources and with different timeliness requirements, enabling each data processing component to be developed, deployed and expanded independently, which greatly improves the overall flexibility and maintainability of the system.

[0036] The customer information search method based on a search engine proposed in this application significantly improves query performance. By pre-integrating multi-source data into a search-optimized index and pre-processing complex calculations, it completely avoids multi-table joins and real-time calculations during queries. This allows it to maintain millisecond-level response speeds even when facing complex queries with tens of millions of data points and dozens of filtering conditions, greatly improving the operational efficiency of business personnel and the user experience.

[0037] Secondly, it achieves precise balance control in terms of data quality. By designing the most suitable synchronization strategy (real-time log capture, real-time stream computing, offline batch processing) for different types of data, it not only ensures the timeliness of core business information and high-frequency behavioral indicators, but also ensures the accuracy of macro statistical indicators and the economy of calculation, thereby comprehensively guaranteeing the reliability of search results and the effectiveness of decision support.

[0038] Finally, in terms of system architecture, it achieves a high degree of loose coupling and scalability, decoupling data synchronization logic from core business systems and integrating them through a unified message pipeline. This allows data source changes, computing logic adjustments, or search engine expansion to be carried out independently, significantly reducing the complexity of long-term system operation and maintenance and the cost of iterative upgrades, laying a solid technical foundation for coping with the continuous growth of enterprise business and data.

[0039] Optionally, in some embodiments of this application, the third type of customer data is calculated in real time based on business event messages, including: Upon receiving a business event message, update the corresponding customer status data based on the customer identifier in the business event message; The third type of customer data is calculated in real time based on the updated status data and stored as a Boolean field corresponding to the index.

[0040] Specifically, the system continuously monitors business event messages from upstream business activities. These messages are essentially standardized notifications of specific actions within the business system (such as "sales completed a customer visit" or "customer submitted a service request"). Once such a message is received, the system first updates the customer's temporary status data in memory or cache based on the customer identifier carried in the message; for example, it records the latest time of the contact. Next, instead of simply storing this raw time, the system performs a lightweight calculation in real time based on preset business rules (such as checking if the difference between the "current time" and the "latest contact time" is greater than 365 days). The result of this calculation is directly converted into a clear Boolean value (True / False), such as "is_contacted_within_365days: false". This Boolean value is then used as an independent index field to update the corresponding customer's document in the target search engine. Here, the "Boolean field" pre-defines a complex condition that would otherwise require date calculations during querying into a discrete value that can be quickly matched in the index. Therefore, when business users execute queries containing the condition "not touched in the last 365 days", the search engine does not need to perform any calculations. It can directly filter the query by using the inverted index of this boolean field, which improves query performance by several orders of magnitude.

[0041] This application embodiment greatly reduces query response latency by shifting the calculation of query conditions from "at query time" to a one-time "at write time," while maintaining the high real-time performance of behavioral indicators, providing technical support for complex real-time decision-making scenarios.

[0042] Optionally, in some embodiments of this application, the method of this application further includes the following steps: Assign a unique tracking identifier to at least a portion of the synchronized data; Regularly sample and compare the consistency between the underlying database or data warehouse and the corresponding data in the target search engine; When an inconsistency is detected, the corresponding data resynchronization or repair task is automatically triggered based on the unique tracking identifier.

[0043] In this embodiment, at the source of data generation or just as it enters the unified message middleware (such as Kafka), the system assigns a unique trace ID to each piece of critical data that needs to be synchronized. This ID acts like a data identity card, accompanying the data through all subsequent stages such as BinLog parsing, real-time computation, and offline task output, and may be recorded in a separate audit log. Subsequently, the system initiates a low-frequency "periodic sampling comparison" task. This task is not a full comparison (too costly), but rather periodically (e.g., every hour) randomly selects a batch of customer IDs and retrieves complete information about the same customer from both the data source (e.g., the business database for the first type of data, and the data warehouse for the third type of data) and the data endpoint (the target search engine) for refined comparison. When this process identifies inconsistencies (e.g., the customer city information in the search database has not been updated), the system can use the previously embedded unique trace ID to quickly locate the specific stage in the synchronization pipeline where the data was lost or erroneous (e.g., a BinLog message was not successfully consumed). Furthermore, the system can automatically trigger targeted resynchronization or repair tasks, such as resending the lost message or recalculating the customer's metrics.

[0044] This mechanism transforms the handling of data consistency issues from passive manual investigation and repair to a proactive, automated operation and maintenance process. It significantly improves system reliability and maintainability, ensuring that temporary network fluctuations or program malfunctions do not cause permanent data inconsistencies. Through automated compensation mechanisms, it guarantees that the data in the search engine ultimately remains consistent with the source system, enhancing the robustness of the entire solution in a production environment.

[0045] In some embodiments of this application, optionally, the target search engine is the Elasticsearch distributed search and analytics engine.

[0046] Elasticsearch, a distributed search engine built on Lucene, boasts two key capabilities: First, its powerful inverted indexing ability enables efficient indexing of fields such as text, numbers, dates, and even booleans, achieving millisecond-level response times for multi-condition filtering and full-text search. This directly meets the stringent performance requirements of the solution. Second, its distributed architecture allows index data to be sharded and stored and processed across multiple nodes, providing near-linear horizontal scalability. In this application, massive data update requests from three data synchronization channels are continuously written to the Elasticsearch cluster, whose distributed nature allows it to efficiently handle this high-throughput write pressure. Furthermore, when faced with a large number of concurrent and complex query requests, the cluster can distribute the query load across multiple nodes for parallel processing and then aggregate the results, ensuring stable query performance under high concurrency scenarios. Therefore, the benefits of using Elasticsearch are twofold: First, its inverted index mechanism is the technical foundation for achieving a fundamental performance leap from complex relational queries to efficient document retrieval; second, its distributed nature provides scalable and robust underlying support for the system to cope with the continuous growth of enterprise customer data volume and the increase in query concurrency.

[0047] In some embodiments of this application, optionally, the step of updating the target search library with the obtained second-type or third-type search data is optimized using batch submission: Multiple single data write operations targeting the target search library are merged into a single batch request; Submit a batch of requests to the target search library for execution at once.

[0048] In some embodiments of this application, each data update instruction from the message listener (such as "update customer A's city field") may trigger a single HTTP API call to Elasticsearch. When data synchronization traffic is massive, this commit-by-commit mode generates significant network round-trip overhead and places a heavy burden on the request processing threads of the Elasticsearch cluster, ultimately leading to a bottleneck in write throughput and increased latency. However, in the embodiments of this application, the system internally (e.g., within the synchronization service) sets up a buffer to temporarily store and merge multiple single update operations received within a short period in memory, assembling them into a batch request containing multiple update instructions. Then, this batch request is submitted to Elasticsearch for execution all at once through a single network connection. For Elasticsearch, processing a batch request containing 100 updates has a much lower internal overhead than processing 100 independent requests. This optimization is applied throughout the synchronization process for all three types of data.

[0049] The embodiments of this application significantly reduce the burden on the search engine to process requests, multiply the write throughput of the data synchronization pipeline, enabling the system to reflect changes in source data in the search index with higher efficiency, while also indirectly reducing synchronization latency and alleviating the pressure on the target search engine cluster, thereby improving the resource utilization efficiency and stability of the entire system.

[0050] In some embodiments of this application, optionally, before updating the third type of customer data to the target search engine, the method further includes: converting the raw indicator values ​​obtained by streaming computation into one or more corresponding Boolean or enumerated pre-computed field values ​​based on one or more predefined query condition templates, and storing these pre-computed fields as independent, indexable fields.

[0051] When a real-time stream processing framework (such as Flink) completes the processing of business event message streams and generates a raw metric value—for example, calculating a specific date and time for customer A's "last sales contact" or an integer value for "number of logins in the last 7 days"—while this data is already a derived metric, it's not the most suitable form for efficient filtering by search engines. This is because the query templates it faces can be patterns abstracted from high-frequency, typical business query scenarios, such as "filtering out customers who haven't been contacted in the last 365 days" or "filtering out highly active customers (defined as: more than 20 logins in the last 30 days)." The system will then reprocess these raw metric values ​​in real time based on these templates.

[0052] Specifically, the system loads these predefined templates, which explicitly define how to map one or more raw metrics into a query-friendly format. For example, for the raw metric "last contact time," a template called "no contact within 365 days" will calculate the difference between the current time and that time in real time and convert the result into a boolean value (True or False). Similarly, for the raw value "number of logins in the last 7 days," a template called "activity level" might convert it into an enumeration value, such as "inactive," "normal," and "active," based on a threshold (e.g., 0 times, 1-5 times, more than 5 times). This conversion process brings forward the dynamic calculation logic that may involve time calculations or numerical comparisons, which will need to be repeatedly executed during future queries, to the moment of data writing and solidifies it.

[0053] These newly generated Boolean or enumerated pre-computed field values ​​do not overwrite or replace the original indicator values. Instead, they are added to the customer data record as independent, new fields, and are prepared to be written to the target search engine along with the original data. From the search engine's perspective, these new fields (such as is_uncontacted_365d: true, activity_level: "active") are no different from other fields, and inverted indexes will be built for them in the same way. Boolean and enumerated fields are among the most efficient data types for search engine filtering because their value range is limited, the index structure is very compact, and nanosecond-level matching speeds can be achieved.

[0054] In this way, when business users initiate a query containing the condition "no contact in the last 365 days," the search engine does not need to perform date function calculations and range comparisons on the "last contact time" field of thousands of records during the query. Instead, it only needs to quickly locate all document IDs with a value of true in the inverted index of the boolean field is_uncontacted_365d. This shift from real-time row calculation to pre-computed column matching reduces query complexity by several orders of magnitude, enabling stable millisecond-level response even when performing the most complex multi-condition filtering on massive datasets. This is the technological cornerstone for achieving efficient interactive analysis. When a new high-frequency query pattern needs to be supported (e.g., adding a filter for "whether it is a new customer (registered within 30 days)"), data engineers only need to add a new transformation rule to the predefined template library, and the stream processing program will automatically start generating and indexing the corresponding new field. The front-end query service does not need to modify complex logic; it only needs to add support for filtering the new field to the interface. This architecture, which decouples business query semantics from underlying data processing, allows the system to respond quickly to business changes, significantly reducing the complexity and cost of iterative development and long-term maintenance.

[0055] The implementation methods and advantages of the embodiments of this application have been described above through multiple examples. The specific processing procedures of the embodiments of this application are described in detail below with reference to specific examples.

[0056] The company's sales team uses a CRM system to query the company's customer database. The basic database contains information on all of the company's customers, with a data volume exceeding ten million. Figure 3 This illustration shows fields included in a basic database according to an embodiment of this application. A sales representative needs to query customers who meet the following criteria: [Shanghai City], [Public Leads Database], [Not listed by other sales representatives], [Excluding customers without phone numbers], [Not contacted by other sales representatives in the last 365 days], [Has had an income greater than 50,000 RMB in at least one of the past 5 years], and [Her HR has been frequently active in the past].

[0057] The above search includes customer-related information (city, location in the public database), the relationship between the salesperson and the customer (whether the customer is listed by the salesperson), the customer's contact information (no phone customers), the customer's visit contact information (no contact by other salespeople in the last 365 days), the customer's past order information (at least one year of income greater than 50,000), and the customer's HR usage information on the platform (HR is frequently active). Finally, the query results have more than 300 fields and need to be displayed according to the salesperson's custom requirements.

[0058] Figure 4 This is a schematic diagram summarizing the overall processing procedure of the customer information search method based on a search engine according to an embodiment of this application. Combined with... Figure 4 As shown, the customer information search process in this application embodiment includes: assembling all the query conditions of the customer query into a JSON array, calling the Elasticsearch query service, querying Elasticsearch, and obtaining the query results.

[0059] For the output results, a RESTful API is provided to return results according to the user-defined query display fields. This allows the P95 to achieve query speeds within 100 milliseconds, significantly improving the user's query experience. Furthermore, any changes to related fields in any business system will be accurately synchronized according to the agreed-upon timeframe, resolving past issues of data inaccuracy in certain scenarios.

[0060] In addition, to ensure the accuracy of data in Elasticsearch during queries, changes to data in the database need to be continuously synchronized to Elasticsearch according to several fixed processes. Specific steps include: Step 1: Divide the data in Elasticsearch into three categories based on data type. The first category is basic customer information. This type of data corresponds one-to-one with the fields of the customer's index record in Elasticsearch. For example, the first three query conditions in the scenario description: customer-related information (city, shared database, relationship between salesperson and customer (whether listed in the salesperson's column), customer contact information (no phone number for customers). The second category is data that requires computation and has low real-time requirements. For example, the last two query conditions in the scenario description: customer's past order information (at least one year's income greater than 50,000), customer's HR usage information on the platform (HR is frequently active). The third category is data that requires computation and has high real-time requirements. For example, the fourth query condition in the scenario description: customer's visit and contact information (not contacted by other salespeople in the last 365 days).

[0061] Step 2: Each type is processed in a different way. For the first type, the basic customer information data can be found here. Figure 5 The first processing path 501 shown uses Datax to parse the BinLog of the database (basic database, including Table 1, Table 2, Table 3... Table N) in real time. It filters the changes of the database fields that need to be processed in the BinLog, sends the processing message to the message middleware KafKa, and the Binlog change handler listens for the message. For each message change, it synchronously calls the Elasticsearch update service to change the data according to certain business rules. The data synchronization latency of this method can be controlled within 100 milliseconds. For the second type, the data requires computation to obtain, and the real-time requirements are not high; see [link to relevant documentation]. Figure 5 The third processing path 503, as shown, involves a data warehouse synchronization task synchronizing data from the database to the data warehouse. The data warehouse has a fixed daily offline metric calculation task, which completes the calculation after T+1 days. After calculation, the data metric is sent to Kafka via a message. Finally, the offline metric synchronization service calls Elasticsearch's update service to synchronize the data according to certain business rules. This method has a data synchronization delay of T+1 days, where T refers to the actual date the data was generated in the business system (e.g., order date, log date, etc.). T+1 indicates that the calculation and synchronization will be completed the day after the data was generated.

[0062] For the third type, the data requires computation to obtain, and the real-time requirements are high; see [link to relevant documentation]. Figure 5 The second processing path 502 shown involves the upstream business application sending a business event message to Kafka when the data changes. The simple metric calculation service listens to this message and calculates the target metric in memory for different business event types. After the calculation is completed, the Elasticsearch update service is called synchronously to change the data according to certain business rules. The data synchronization latency of this method can be controlled within 100 milliseconds. Within the Elasticsearch synchronization service, tens of thousands of data synchronization write tasks may be processed in a short period of time. To improve performance and reduce the performance overhead on Elasticsearch, we adopted batch commit optimization: multiple write operations are merged into batch requests and submitted to Elasticsearch, which greatly improves the efficiency of data writing.

[0063] Furthermore, to improve query efficiency, dedicated pre-calculated fields are created for the results of certain information fields (highly real-time computational data). In the "Simple Indicator Calculation Service," not only are indicator values ​​calculated, but the results are also transformed into a more search-friendly format. For example, after calculating "Recent Contact Time," a Boolean field for "Whether there has been no contact within the last 365 days" is simultaneously generated and indexed. This improves the performance of such queries by several orders of magnitude.

[0064] Additionally, when designing the index, explicitly define each field to prevent Elasticsearch from inaccurately inferring the type automatically, and set fields that contain long text and are not included in the search to not be indexed.

[0065] To ensure data consistency, the method in this application also includes an asynchronous data consistency verification and repair service as an independent monitoring process. Specifically, when data enters Kafka via messages, each data item is assigned a unique trace ID, which flows through the entire pipeline. The "source information" and "current processing status" of this data item are recorded in a separate log database. The verification service periodically (e.g., hourly) randomly samples a batch of customer IDs, retrieves the complete data for that customer from the source business database and Elasticsearch, and compares them. When inconsistencies are found, the cause of the inconsistency can be traced based on the trace ID, and the corresponding data pipeline retry or data repair task can be automatically triggered. For example, if some basic information is found to be out of sync, the corresponding Binlog change message is automatically resent.

[0066] In summary, the customer information retrieval method based on a search engine proposed in this application has the following beneficial effects: (1) Significantly improved query efficiency. By constructing an architecture that separates the search engine from the business database, complex queries are offloaded from the traditional database, and the inverted index mechanism of the search engine is used to achieve millisecond-level retrieval of customer information.

[0067] (2) Data consistency is guaranteed. A data pipeline combining real-time synchronization based on database log parsing and offline batch processing is adopted to ensure the eventual consistency of customer information across multiple systems.

[0068] (3) Strong real-time data synchronization. By parsing the binary log of the database and using a message queue for asynchronous transmission, the system achieves second-level synchronization of customer information changes, ensuring the timeliness of search results.

[0069] (4) Excellent system scalability. The system adopts a distributed search engine architecture, which can linearly increase the system's processing capacity by adding nodes, meeting the needs of enterprise customers for continuous growth in data volume.

[0070] (5) The data model is optimized and reasonable. By constructing a search-oriented wide table data model, multi-table join queries are transformed into single-index retrieval, which greatly reduces query complexity.

[0071] The method in this application effectively solves the problems of performance bottlenecks and data inconsistencies in traditional customer information queries, and provides an efficient and reliable search solution for systems such as customer relationship management.

[0072] Correspondingly, this application also provides a customer information search device based on a search engine, see reference. Figure 6 ,include: The parsing module 110 is used to capture change events of the first type of customer data in real time by parsing the binary log of the basic database, and send the change events to the message middleware. The first type of customer data includes the basic attribute information of the customer. The basic database is used to store customer information generated by upstream business activities in the form of multiple data tables. The sending module 120 is used to send business event messages generated by upstream business activities to the message middleware. The business event message includes the business event type, the customer identifier and personnel identifier of the business event participants, and the time when the business event occurred. The acquisition module 130 acquires a second type of customer data based on customer information in the data warehouse and sends the second type of customer data to the message middleware; wherein, the second type of customer data is customer statistical indicators obtained through offline batch calculation, including at least one of the following: whether the customer's historical revenue meets preset conditions and the customer's order behavior indicators within a preset historical period; The monitoring module 140 monitors the message middleware. If the message middleware contains change messages related to the first type of customer data or the second type of customer data, it updates the index field of the corresponding customer document in the target search engine based on the change messages to achieve real-time synchronization of the first type of customer data or the second type of customer data. If the message middleware contains the business event messages, it generates third type of customer data based on the business event messages using real-time stream computing and updates the index field of the corresponding customer document in the target search engine. The third type of customer data consists of customer behavior indicators that need to be calculated in real time, including at least one of the following: whether the customer has been contacted by sales within a preset time period, and the customer's activity level within a preset period. The retrieval module 150 is used to respond to a customer information query request initiated by a user, retrieve the index in the target search engine and return the search results, wherein the query request contains at least one query condition.

[0073] Based on at least one of the above embodiments, the electronic device in the embodiments of this application may be a user terminal device, a server, other computing devices, or a cloud server. Figure 7 The diagram illustrates the hardware structure of an electronic device according to an embodiment of this application. The electronic device may include a processor 601 and a memory 602 storing computer program instructions. The processor 601 reads and executes the computer program instructions stored in the memory 602 to implement the process or function of any of the methods described above.

[0074] Specifically, processor 601 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. Memory 602 may include a mass storage device for data or instructions. For example, memory 602 may be at least one of the following: a hard disk drive (HDD), read-only memory (ROM), random access memory (RAM), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, universal serial bus (USB) drive, or other physical / tangible memory storage device. Alternatively, memory 602 may include removable or non-removable (or fixed) media. Furthermore, memory 602 may be internal or external to the integrated gateway disaster recovery device. Memory 602 may be non-volatile solid-state memory. In other words, typically memory 602 includes a tangible (non-transitory) computer-readable storage medium (such as a memory device) encoded with computer-executable instructions, and when the software is executed (e.g., by one or more processors), it can perform the operations described in the methods of the embodiments of this application.

[0075] As an example, Figure 7The illustrated electronic device may also include a communication interface 603 and a bus 610. The processor 601, memory 602, and communication interface 603 are connected via bus 610 and communicate with each other. Bus 610 may include hardware, software, or both, and may couple components of an online data traffic metering device together. The bus may include at least one of the following: Accelerated Graphics Port (AGP) or other graphics bus, Enhanced Industry Standard Architecture (EISA) bus, Front Side Bus (FSB), HyperTransport (HT) Interconnect, Industry Standard Architecture (ISA) bus, Infinite Bandwidth Interconnect, Low Pin Count (LPC) bus, memory bus, Microchannel Architecture (MCA) bus, Peripheral Component Interconnect (PCI) bus, PCI-Express (PCI-X) bus, Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association Local (VLB) bus, or other suitable bus. Bus 610 may include one or more buses. Although specific buses are described or shown in the embodiments of this application, any suitable bus or interconnection method is contemplated in the embodiments of this application.

[0076] In conjunction with the methods in the above embodiments, this application also provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the process or function of any of the methods in the above embodiments.

[0077] This application also provides a computer program product that stores computer program instructions, which, when executed by a processor, implement the process or function of any of the methods described above.

[0078] The flowcharts and / or block diagrams of methods, terminals, systems, and computer program products according to embodiments of this application have been exemplarily described above, and related aspects have been described. It should be understood that each block or combination thereof in the flowcharts and / or block diagrams may be implemented by computer program instructions, by dedicated hardware performing a specified function or action, or by a combination of dedicated hardware and computer instructions. For example, these computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to form a machine such that these instructions, executed via such processor, enable the implementation of the function / action specified in each block or combination thereof in the flowcharts and / or block diagrams. Such a processor may be a general-purpose processor, a dedicated processor, a special-purpose application processor, or a field-programmable logic circuit.

[0079] The functional blocks shown in the structural block diagrams of this application can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc.; when implemented in software, they are programs or code segments used to perform the required tasks. Programs or code segments can be stored in memory or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. Code segments can be downloaded via computer networks such as the Internet or intranets.

[0080] It should be noted that this application is not limited to the specific configurations and processes described above or shown in the figures. The above descriptions are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the described systems, devices, terminals, modules, or units can be referred to the corresponding processes in the method embodiments, and need not be repeated here. It should be understood that the scope of protection of this application is not limited thereto. Any equivalent modifications or substitutions that can be conceived by those skilled in the art within the scope of the technology disclosed in this application should be covered within the scope of protection of this patent application.

Claims

1. A customer information retrieval method based on a search engine, characterized in that, The method includes: By parsing the binary logs of the basic database, change events of the first type of customer data are captured in real time, and the change events are sent to the message middleware. The first type of customer data includes the basic attribute information of customers. The basic database is used to store customer information generated by upstream business activities in the form of multiple data tables. The business event message generated by the upstream business activity is sent to the message middleware. The business event message includes the business event type, the customer identifier and personnel identifier of the business event, and the time when the business event occurred. The second type of customer data is obtained based on customer information in the data warehouse, and the second type of customer data is sent to the message middleware; wherein, the second type of customer data is customer statistical indicators obtained through offline batch calculation, including at least one of the following: whether the customer's historical revenue meets preset conditions, and the customer's order behavior indicators within a preset historical period; If the message middleware is monitored and there is a change message related to the first type of customer data or the second type of customer data in the message middleware, the index field of the corresponding customer document in the target search engine is updated based on the change message to achieve real-time synchronization of the first type of customer data or the second type of customer data. If the message middleware contains the business event message, based on the business event message, a third type of customer data is generated through real-time stream computing, and the index field of the corresponding customer document in the target search engine is updated; wherein, the third type of customer data is a customer behavior indicator that needs to be calculated in real time, including at least one of whether the customer has been contacted by sales within a preset time period and the customer's activity index within a preset period. In response to a user-initiated customer information query request, the system retrieves the index in the target search engine and returns the search results, wherein the query request contains at least one query condition.

2. The method according to claim 1, characterized in that, The real-time calculation of the third type of customer data based on the business event message includes: Upon receiving a business event message, update the corresponding customer status data based on the customer identifier in the business event message; The third type of customer data is calculated in real time based on the updated status data and stored in the form of a boolean field corresponding to the index.

3. The method according to claim 1, characterized in that, Also includes: Assign a unique tracking identifier to at least a portion of the synchronized data; Periodically sample and compare the consistency of the basic database or data warehouse with the corresponding data in the target search engine; When an inconsistency is detected, the corresponding data resynchronization or repair task is automatically triggered based on the unique tracking identifier.

4. The method according to claim 1, characterized in that, The target search engine is the Elasticsearch distributed search and analytics engine.

5. The method according to claim 1, characterized in that, In the step of updating the target search database with the obtained second or third type of customer data, batch submission optimization is used: Multiple single data write operations targeting the target search library are merged into a single batch request; The batch requests are submitted to the target search engine for execution at once.

6. The method according to claim 1, characterized in that, In some embodiments of this application, optionally, before updating the third type of customer data to the target search engine, the method further includes: converting the raw indicator values ​​obtained by streaming computation into one or more corresponding Boolean or enumerated pre-computed field values ​​based on one or more predefined query condition templates, and storing these pre-computed fields as independent, indexable fields.

7. A customer information search device based on a search engine, characterized in that, include: The parsing module is used to capture change events of the first type of customer data in real time by parsing the binary log of the basic database, and send the change events to the message middleware. The first type of customer data includes the basic attribute information of the customer. The basic database is used to store customer information generated by upstream business activities in the form of multiple data tables. The sending module is used to send business event messages generated by upstream business activities to the message middleware. The business event message includes the business event type, the customer identifier and personnel identifier of the participating business event, and the time when the business event occurred. The acquisition module acquires a second type of customer data based on customer information in the data warehouse and sends the second type of customer data to the message middleware; wherein, the second type of customer data is customer statistical indicators obtained through offline batch calculation, including at least one of the following: whether the customer's historical revenue meets preset conditions and the customer's order behavior indicators within a preset historical period; The monitoring module monitors the message middleware. If the message middleware contains change messages related to the first type of customer data or the second type of customer data, it updates the index field of the corresponding customer document in the target search engine based on the change messages to achieve real-time synchronization of the first type of customer data or the second type of customer data. If the message middleware contains the business event messages, it generates third type of customer data based on the business event messages using real-time stream computing and updates the index field of the corresponding customer document in the target search engine. The third type of customer data consists of customer behavior indicators that need to be calculated in real time, including at least one of the following: whether the customer has been contacted by sales within a preset time period, and the customer's activity level within a preset period. The retrieval module is used to respond to a customer information query request initiated by a user, retrieve the index in the target search engine and return the search results, wherein the query request contains at least one query condition.

8. An electronic device, characterized in that, The electronic device is a terminal device or a server. The electronic device includes a processor and a memory storing computer program instructions. When the electronic device executes the computer program instructions, it implements the method as described in any one of claims 1-6.

9. A computer program product, characterized in that, It includes computer program instructions that, when executed, implement the method as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, It stores computer program instructions that, when executed, implement the method as described in any one of claims 1-6.

Images