Systems, platforms, and methods for data integration based on real-time data streams

Abstract

The application provides a system, platform and method for data integration based on real-time data flow, the system comprising a collector configured to generate a real-time data flow in response to a first data source and collect the real-time data flow; wherein the first data source is one of a plurality of data sources, and the real-time data flow is generated based on user operation; a data flow processor configured to calculate a real-time score of the first data source based on real-time data of each evaluation index extracted from the real-time data flow; obtain a historical score calculated based on historical data flow of a second data source, and calculate a real-time user experience evaluation score based on the real-time score and the historical score; and a data flow outputter configured to output the user experience evaluation score in different output formats according to different output positions. The application can be used to trigger the calculation and analysis of the user experience evaluation score immediately after submitting feedback or performing business operations, thereby improving response speed and user experience.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a system, platform and method for data integration based on real-time data streams. Background Technology

[0002] The continuous development of data processing technologies has provided various enterprises and organizations with a wealth of data analysis tools and methods. Based on these tools and methods, enterprises and organizations can achieve data integration and synchronization, thereby enabling business decision-making and operational optimization. Data integration is the process of combining data from different data sources, while data synchronization refers to synchronizing data from one database or system to another to ensure data consistency and real-time performance.

[0003] Generally, data integration and synchronization solutions are non-real-time. Non-real-time data integration and synchronization solutions mainly rely on batch processing. This method usually collects data at preset time intervals (such as daily or hourly) and processes the data in batches after the batch ends in order to capture user dynamics.

[0004] It is evident that non-real-time data integration and synchronization solutions, which analyze and process data in batches, have a certain delay. This delay may lead to decision-making lag and resource waste, making it difficult to meet the real-time monitoring needs of enterprises or organizations. Summary of the Invention

[0005] This application provides a system, platform, and method for data integration based on real-time data streams to address the latency issues inherent in traditional data integration methods.

[0006] In a first aspect, embodiments of this application provide a system for data integration based on real-time data streams, comprising: a collector configured to collect real-time data streams in response to a first data source generating such a stream; wherein the collector includes multiple types, with different types of collectors corresponding to different types of data sources, the first data source being one of multiple data sources, and the real-time data stream being generated based on user operations; a data stream processor configured to calculate a real-time score for the first data source based on real-time data of various evaluation indicators extracted from the real-time data stream; and to acquire historical scores calculated from historical data streams for a second data source, and to calculate a real-time user experience evaluation score based on the real-time score and the historical score, wherein the second data source is any data source other than the first data source among multiple data streams; the data source includes at least one of a business system, a CRM system, an online survey system, an online customer service and feedback system, and an external data source; and a data stream outputter configured to output the user experience evaluation score in different output formats depending on the output location, including a database, a file system, or an FTP server.

[0007] In one possible implementation, the data stream outputter is specifically configured to: output user experience evaluation scores to a historical record database to update the historical scores corresponding to the first data source; and the data stream outputter is further configured to: determine whether the user experience evaluation score is lower than a preset threshold; if the user experience evaluation score is lower than the preset threshold, output the user experience evaluation score to a dissatisfied user warning database.

[0008] In one implementation, the data stream processor includes a score calculation processor; the score calculation processor is specifically configured to calculate a user experience evaluation score according to a first formula.

[0009] The first formula is: User Experience Evaluation Score = w1·Business System Score + w2·Online Customer Service and Feedback System Score + w3·Online Survey System Score + w4·External Data Source Score + w5·CRM System Score, where w1, w2, w3, w4 and w5 are the score weights.

[0010] In one feasible approach, the score calculation processor is specifically configured to: extract real-time data for each evaluation indicator from a real-time data stream; wherein different types of data sources correspond to different evaluation indicators, and each data source corresponds to at least one evaluation indicator; and perform weighted summation of the real-time data according to the preset weights of each evaluation indicator to obtain a real-time score.

[0011] In one possible implementation, the data stream processor includes a speech-to-text processor and an analysis and classification processor, and the collector includes a text collector and a speech collector; the text collector is configured to: collect the real-time text data stream generated in response to an online customer service and feedback system, and input the real-time text data stream to the analysis and classification processor; the speech collector is configured to: collect the real-time speech data stream generated in response to an online customer service and feedback system, and input the real-time speech data stream to the speech-to-text processor; the speech-to-text processor is configured to: convert the real-time speech data stream into text data, and input the text data to the analysis and classification processor; the analysis and classification processor is specifically configured to: perform classification processing on the real-time text data stream and / or text data to obtain classification processing results, and input the classification processing results to a score calculation processor so that the score calculation processor calculates a real-time score based on the classification processing results; the classification processing results include classification labels for the real-time text data stream and the text data, the classification labels include business classification labels, tendency analysis labels, and / or keyword labels, and the tendency analysis labels include positive labels, neutral labels, negative labels, and severely negative labels.

[0012] In one feasible approach, when the primary data source is an online customer service and feedback system, the evaluation metrics include initial response time, problem resolution rate, and feedback sentiment analysis; when the primary data source is a business system, the collector is a database collector, and the evaluation metrics include browsing time, business processing frequency, total business value, user operation results, and service evaluation; when the primary data source is a CRM system, the collector is a database collector, and the evaluation metrics include basic user information, behavioral data, and preference data; when the primary data source is an online survey system, the collector is a message queue collector, and the evaluation metrics include survey satisfaction and net promoter score (NPS); when the primary data source is an external data source, the collector is an API interface collector, and the evaluation metrics include industry average service satisfaction and industry average response time.

[0013] Secondly, embodiments of this application provide a data stream integration platform, including a data stream task designer. The data stream task designer is configured to: create a task template in response to user interaction; add multiple task nodes to the task template and establish connection relationships between the task nodes, wherein the task nodes include at least a collector node, a processor node, and an output node; configure a task data stream specification for the task template, wherein the task data stream specification defines the field format of the data stream; and construct a user analysis model based on the task template. The user analysis model is used to run the system for data integration based on real-time data streams in the aforementioned first aspect and its various implementations.

[0014] In one possible implementation, a task runtime controller is also included, which is configured to: create task instances based on task templates in response to a user analytics model, and manage task instances based on task templates and task data flow specifications.

[0015] Thirdly, embodiments of this application provide a method for data integration based on real-time data streams, comprising: in response to a first data source generating a real-time data stream, acquiring the real-time data stream; wherein the first data source is one of multiple data sources, and the real-time data stream is generated based on user operations; calculating a real-time score for the first data source based on real-time data of various evaluation indicators extracted from the real-time data stream; and obtaining a historical score calculated from a historical data stream for a second data source, calculating a real-time user experience evaluation score based on the real-time score and the historical score, wherein the second data source is any data source other than the first data source among multiple data streams; the data source includes at least one of a business system, a CRM system, an online survey system, an online customer service and feedback system, and an external data source; and outputting the user experience evaluation score in different output formats according to different output locations, including databases, file systems, or FTP servers.

[0016] Fourthly, embodiments of this application provide a data stream integration method, comprising: creating a task template in response to a user's interactive action; adding multiple task nodes to the task template and establishing connection relationships between the task nodes, wherein the task nodes include at least a collector node, a processor node, and an output node; configuring a task data stream specification for the task template, wherein the task data stream specification defines the field format of the data stream; and constructing a user analysis model based on the task template, wherein the user analysis model is used in the system for data integration based on real-time data streams in the aforementioned first aspect and its various implementations.

[0017] As described above, this application provides a system, platform, and method for data integration based on real-time data streams. The system includes a collector configured to collect real-time data streams in response to a first data source. The collectors can be of various types, each corresponding to a different data source. The first data source is one of multiple data sources, and the real-time data stream is generated based on user actions. A data stream processor is configured to calculate the real-time score of the first data source based on real-time data of various evaluation indicators extracted from the real-time data stream. It also acquires historical scores calculated from historical data streams of a second data source, and calculates a real-time user experience evaluation score based on the real-time score and historical scores. The second data source includes data sources other than the first data source among multiple data streams. Data sources include at least one of a business system, CRM system, online survey system, online customer service and feedback system, and external data sources. A data stream outputter is configured to output the user experience evaluation score in different output formats depending on the output location, including a database, file system, or FTP server. The system provided by this application can be used to immediately trigger the calculation and analysis of user experience evaluation scores after submitting feedback or performing business transactions, thereby improving response speed and user experience. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the structure of the data stream integration platform provided in the embodiments of this application;

[0019] Figure 2 This is a schematic diagram of the structure of the user analysis model provided in the embodiments of this application;

[0020] Figure 3 This is a schematic diagram of the structure of a system for data integration based on real-time data streams, provided in an embodiment of this application.

[0021] Figure 4 A flowchart illustrating a method for data integration based on real-time data streams provided in an embodiment of this application;

[0022] Figure 5This is a flowchart illustrating the data flow integration method provided in an embodiment of this application.

[0023] Among them, 100-Dataflow Task Designer; 200-Task Runtime Controller; 300-Collector; 301-Database Collector; 302-Message Queue Collector; 303-File Collector; 3031-Text Collector; 3032-Voice Collector; 304-API Interface Collector; 400-Dataflow Processor; 401-Score Calculation Processor; 402-Speech-to-Text Processor; 403-Analysis and Classification Processor; 500-Dataflow Outputter. Detailed Implementation

[0024] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of this application.

[0025] To facilitate understanding of the technical solutions of the embodiments of this application, before describing the specific implementation methods of the embodiments of this application, some technical terms in the technical field to which the embodiments of this application belong will be briefly explained. Among them:

[0026] Real-time data stream processing: This involves processing and analyzing data generated in real time to quickly obtain information and respond. Unlike non-real-time batch processing solutions, real-time data stream processing enables low latency and continuous processing, making it suitable for scenarios requiring real-time response and dynamic decision-making, such as financial monitoring, online recommendation, and IoT analytics.

[0027] This application provides a system, platform, and method for data integration based on real-time data streams, utilizing real-time data processing technology. This system can collect and analyze multi-source data such as customer feedback, transaction records, and social media interactions in real time to promptly capture changes in customer sentiment and fluctuations in satisfaction. This not only improves the company's responsiveness to customer needs but also helps the company better predict potential problems, formulate more precise customer service strategies, and ultimately enhance customer satisfaction and loyalty. In specific implementation, the system provided in this application can immediately trigger the calculation and analysis of user experience evaluation scores after a user submits feedback or performs business transactions. The system can identify potentially dissatisfied users in real time and quickly initiate subsequent user care and relationship repair measures, effectively resolving user dissatisfaction and reducing the risk of churn. In contrast, batch processing-based models typically require a day or longer to identify potentially dissatisfied users, easily missing opportunities for timely intervention, leading to increased dissatisfaction, which in turn increases the risk of user churn and reduces overall satisfaction.

[0028] Specifically, this application provides a system for data integration based on real-time data streams. This system can run a user analysis model to achieve dynamic monitoring of user satisfaction. The user analysis model can be built on a data stream integration platform. The data stream integration platform will be described in detail below.

[0029] Figure 1 This is a schematic diagram of the structure of the data flow integration platform provided in the embodiments of this application.

[0030] like Figure 1 As shown, the data flow integration platform provided in this application embodiment may include a data flow task designer 100 and a task runtime controller 200.

[0031] The dataflow task designer 100 can be web-based and accessed via a web browser. It provides a graphical interface for users to easily create, manage, and optimize dataflow integration tasks. The designer includes multiple components such as a toolbox, properties window, workspace, and menu bar. Users can add and configure nodes from the toolbox via drag-and-drop, and set detailed parameters for nodes (e.g., collector nodes, processor nodes, output nodes) in the properties window. Furthermore, users can arrange and connect nodes in the workspace, defining execution conditions and logic between nodes through configurable connections. This intuitive design approach not only lowers the technical barrier but also improves design efficiency and task accuracy, enabling users to easily construct complex dataflow integration tasks.

[0032] The task runtime controller 200 can be used for the execution management of data stream integration tasks, specifically including the execution and logical judgment of each node within the task. For example, the task runtime controller 200 can be used for task triggering, process scheduling, resource allocation, and runtime monitoring. The task runtime controller 200 can ensure that each node runs correctly according to preset order and conditions, handling all stages of data acquisition, processing, and output. The task runtime controller 200 can also include a timed triggering function, which can automatically start the task according to a predetermined plan, ensuring that the data stream integration task runs within a specified time or interval. Furthermore, the task runtime controller 200 can monitor the execution status of the task, providing logging and error handling mechanisms to ensure the reliability and stability of the task. Through effective resource allocation and optimization, the task runtime controller 200 can improve the execution efficiency of the task, meeting the needs of high concurrency and large data volume processing.

[0033] Furthermore, in this embodiment, the data stream integration task can define a data stream integration process, which may include two parts: a task data stream specification and a task flow. The task data stream specification defines the data stream fields and formats within the task, ensuring data consistency and readability throughout the integration process. The task flow can consist of multiple nodes, which can be collectors, processors, and outputters, responsible for collecting, processing, and outputting task data, respectively. Nodes can be connected by lines, indicating the order of execution. These connections not only define the connection relationships between nodes but can also configure execution conditions to meet the logical requirements of different scenarios. Through this structured definition, the data stream integration task can efficiently and flexibly manage and execute complex data processing flows, ensuring accurate data transmission and processing between nodes.

[0034] Furthermore, in data stream integration tasks, collector nodes can be used to connect to various data sources, monitor data changes in these data sources in real time, and collect data streams. Collector nodes can handle multiple types of data sources, including databases, file systems, and message queues. Collector nodes can detect and capture data addition, update, and deletion operations, and can convert these changes into standardized data formats to ensure data consistency and integrity during transmission. Through efficient collection mechanisms, collector nodes can quickly respond to data changes, providing timely and accurate data input for subsequent data processing, analysis, and storage. In data stream integration tasks, collector nodes can also be called data stream collectors. Collector nodes can include file collectors, message queue collectors, and database collectors.

[0035] A file collector is a type of data stream collector used to monitor specified local or remote file directories. It automatically detects newly added files within the directory, captures these files, parses their contents into a data stream, and passes it to subsequent processing nodes. File collectors support parsing various file types and formats and can preprocess files according to configuration, such as filtering, splitting, and transforming them. By continuously monitoring the file directory, file collectors ensure the real-time nature and continuity of data, providing timely and accurate data input for subsequent data processing and analysis. Whether it's log files, sensor data, or other forms of file input, file collectors can efficiently transform file content into a processable data stream, integrating it into the overall data processing workflow.

[0036] A message queue collector is a type of data stream collector used to monitor specified message queues. Specifically, it automatically detects new message data in the queue, captures these messages, parses their content into a data stream, and passes it to subsequent processing nodes. Message queue collectors support various message queue systems, such as Apache Kafka and RabbitMQ. By continuously monitoring message queues, they ensure data real-time performance and continuity, providing timely and accurate data input for subsequent data processing and analysis.

[0037] A database collector is a type of data stream collector used to monitor specified database tables. Specifically, it automatically detects new data in a table, captures this data, parses its content into a data stream, and passes it to subsequent processing nodes. Database collectors support various database systems, such as MySQL, PostgreSQL, Oracle, and SQL Server. By continuously monitoring database tables, database collectors ensure the real-time nature and continuity of data, providing timely and accurate data input for subsequent data processing and analysis.

[0038] Furthermore, the data flow task designer 100 is specifically used to perform the following steps S101-S104.

[0039] S101: Create a task template in response to user interaction.

[0040] Understandably, users can interact with the data flow task designer 100 through the graphical interface of the workspace. For example, task templates may include task template 1 and task template 2.

[0041] S102: Add multiple task nodes to the task template and establish the connection relationship between each task node. The task nodes include at least a collector node, a processor node, and an output node.

[0042] For example, task template 1 may include the following nodes: collector-file, processor 1, processor 2, output-database, and task template 2 may include the following nodes: collector-database, processor 1, processor 2, output-FTP, and output-message queue. The database collector can input its collected data stream to processor 1 under condition A and input its collected data stream to processor 2 under condition B. The output-FTP is to output data to a File Transfer Protocol (FTP) server.

[0043] S103: Configure the task data flow specification for the task template, whereby the task data flow specification defines the field format of the data flow.

[0044] In practical applications, task data flow specifications can be configured based on actual needs; no specific limitations are made here.

[0045] S104: Build a user analysis model based on task templates. The user analysis model is used to run a system that integrates data based on real-time data streams.

[0046] Furthermore, the task runtime controller 200 can be configured to perform the following steps S200: create task instances based on task templates in response to user analysis models, and manage task instances based on task templates and task data flow specifications.

[0047] Task instances may include Task Instance 1 and Task Instance 2, which are only illustrative examples here.

[0048] In this embodiment, a user analysis model can be designed and constructed based on a data flow integration platform. This model enables efficient integration and analysis of enterprise user satisfaction-related data. By building a dynamic and flexible data flow processing architecture, customer behavior and feedback can be captured in real time, providing accurate customer satisfaction analysis and helping enterprises respond quickly to customer needs and improve customer experience.

[0049] Figure 2 This is a schematic diagram of the structure of the user analysis model provided in the embodiments of this application.

[0050] Figure 3 This is a schematic diagram of the structure of a system for data integration based on real-time data streams, provided in an embodiment of this application.

[0051] like Figure 2 and Figure 3 As shown in the figure, this application provides a system for data integration based on real-time data streams. The system may include a collector 300, a data stream processor 400, and a data stream output device 500.

[0052] The collector 300 is configured to perform the following steps S301: in response to the generation of a real-time data stream by a first data source, the collector 300 is collected; wherein the collector 300 includes multiple types, and different types of collectors 300 correspond to different types of data sources, the first data source is one of multiple data sources, and the real-time data stream is generated based on user operations.

[0053] Specifically, the data source is determined based on the actual operation of the enterprise. In response to the enterprise's user satisfaction management needs, the data in the data source can include three types: transaction and behavior data, customer feedback data, and external data. Transaction and behavior data can come from business systems and customer relationship management (CRM) systems. Customer feedback data can come from online survey systems, online customer service and feedback systems, and external data can come from external data sources.

[0054] Business systems are systems involved in enterprise operations, such as sales management systems and e-commerce systems. These systems record key information such as customer purchasing behavior, transaction records, order status, and payment information, providing the enterprise with foundational data for customer behavior analysis. In practical applications, the system provided in this embodiment can obtain key information such as user browsing history, business processing records, and service evaluations from the enterprise's business systems. Specifically, the database collector 301 can monitor data tables such as processing record tables and rating tables in the business system in real time. This allows for automatic detection of new data in these tables and timely capture of this information. The database collector 301 can parse the captured data into a data stream, ensuring that this data can be acquired and processed by the model in real time, providing reliable data support for subsequent analysis.

[0055] A CRM system can be used as a platform to collect and manage customer information, such as customer name, age, gender, contact information, and VIP level. This data helps businesses better understand customer needs and preferences, thereby enabling them to develop more precise marketing strategies and customer relationship management measures. Specifically, a database collector like 301 can be used to monitor the user basic information table in the CRM system in real time, automatically detecting new or modified data in the table, capturing and parsing this data into a data stream for collection.

[0056] Online survey systems are tools used by businesses to collect customer feedback and opinions. They can utilize various methods such as questionnaires, scales, and polls to allow participants to engage in the survey process and obtain direct feedback on product usage experience, service quality, brand image, customer satisfaction, and loyalty. Specifically, the system in this embodiment can collect user survey information from the online survey system, including key data such as survey satisfaction and Net Promoter Score (NPS) (whether a user is willing to recommend the company or product to others). This information can be collected and processed in real time through the message queue collector 302.

[0057] In some implementations, the data source can also include a complaint management system, which can be a type of online survey system. After generating a new complaint record, the complaint management system pushes it to a Kafka queue. The message queue collector 302 can continuously monitor this queue, automatically capture new complaint data, and immediately convert it into a data stream for real-time analysis and processing by the model.

[0058] Online customer service and feedback systems are channels provided by businesses to customers, which can include online chat, telephone support, email, etc. Customers raise questions through these channels, and customer service personnel answer or provide solutions based on the type of question. Online customer service systems can record detailed content of each interaction, including customer questions, solutions, and emotional feedback during the interaction. This data can help businesses identify potential problems customers may encounter when using products or services and provide a basis for further product and service optimization. Specifically, this application can obtain user feedback information from the online customer service and feedback system, including text feedback and audio recordings. The file collector 303 monitors the directory of text files and audio recordings of feedback information in real time and automatically captures newly added files.

[0059] External data originates from various information channels outside the enterprise, including market research reports, industry analyses, social media comments, and competitor activities. Market research reports, collected and compiled by professional research institutions, typically include information on industry trends, consumer behavior, and market share. This data helps enterprises understand the overall market situation, potential consumer needs, and industry development trends, thereby enabling them to make beneficial adjustments in strategy formulation. Specifically, this application can obtain industry average data from external data sources, including information such as service satisfaction and problem response time. Through the API interface collector 304, the API interface of the external data source can be called to capture data information in a timely manner. The collector parses the captured data into a data stream, ensuring that this data can be acquired and processed by the model in real time, providing reliable data support for subsequent analysis.

[0060] It is understandable that each data source can correspond to a collector with 300 nodes, and each data source can correspond to a data stream processor with 400 nodes.

[0061] Furthermore, the data stream processor 400 can be configured to perform the following steps S302: calculate the real-time score of the first data source based on the real-time data of various evaluation indicators extracted from the real-time data stream; and obtain the historical score calculated from the historical data stream for the second data source, and calculate the real-time user experience evaluation score based on the real-time score and the historical score. The second data source is a data source other than the first data source among multiple data streams, including at least one of the following: business system, CRM system, online survey system, online customer service and feedback system, and external data source.

[0062] Historical scores can be stored in a historical record database. It's understandable that the first data source is a data source that generates real-time data streams, while the second data source is a data source that hasn't yet generated new real-time data streams.

[0063] In this embodiment, each data source can correspond to a data stream processor 400. The data stream processor 400 corresponding to the CRM system is used to calculate the CRM system score, the data stream processor 400 corresponding to the online survey system is used to calculate the survey system score, the data stream processor 400 corresponding to the online customer service and feedback system is used to calculate the feedback system score, the data stream processor 400 corresponding to the business system is used to calculate the business system score, and the data stream processor 400 corresponding to the external data source is used to calculate the external data source score. Furthermore, the real-time score corresponding to the first data source can be summed with the historical scores corresponding to each of the second data sources to obtain the final user experience evaluation score.

[0064] Furthermore, the data stream outputter 500 can be used to perform the following step S303: outputting the user experience evaluation score in different output formats according to different output locations, including databases, file systems, or FTP servers. Thus, the system provided in this embodiment can be used to immediately trigger the calculation and analysis of the user experience evaluation score after submitting feedback or performing business transactions.

[0065] Furthermore, the data stream output device 500 is specifically configured to perform the following steps S3031.

[0066] S3031: Output the user experience evaluation score to the historical record database to update the historical score corresponding to the first data source.

[0067] This serves as a backup.

[0068] Furthermore, the data stream output device 500 is also configured to perform the following steps S3032-S3032.

[0069] S3032: Determine whether the user experience evaluation score is lower than the preset threshold t1.

[0070] The preset threshold t1 can be 60 points, and this embodiment does not specifically limit it. It is understood that if the user experience evaluation score is lower than the preset threshold t1, it indicates that the user's dissatisfaction is relatively high.

[0071] S3033: If the user experience evaluation score is lower than the preset threshold t1, the user experience evaluation score will be output to the dissatisfied user warning database.

[0072] In some implementations, the data stream outputter 500 may specifically include an outputter that outputs user dissatisfaction rating records, as well as an outputter that outputs to a database of potential dissatisfying users.

[0073] In this way, the data in the dissatisfied user warning database can be used by the customer care system. Understandably, if the user experience evaluation score is greater than or equal to the preset threshold t1, it does not need to be output to the dissatisfied user warning database.

[0074] Furthermore, the data stream processor 400 may include a score calculation processor 401, which is specifically configured to perform the following step S401: calculate a user experience evaluation score according to a first formula.

[0075] The first formula is: User Experience Evaluation Score = w1·Business System Score + w2·Online Customer Service and Feedback System Score + w3·Online Survey System Score + w4·External Data Source Score + w5·CRM System Score, where w1, w2, w3, w4 and w5 are the score weights.

[0076] As can be seen, the user experience evaluation score is calculated by weighted summation of various data sources. The specific values ​​of w1, w2, w3, w4 and w5 can be designed based on actual needs, and this application embodiment does not impose specific limitations on them.

[0077] Understandably, the values ​​of the business system score, online customer service and feedback system score, online survey system score, external data source score, and CRM system score in the first formula depend on whether the business system, online customer service and feedback system, online survey system, external data source, and / or CRM system generate real-time data streams. If a real-time data stream is generated, the score is equal to the real-time score calculated in real time. If no real-time data stream is generated, the score is equal to the historical score corresponding to that data source.

[0078] The score calculation processor 401 is also configured to perform the following steps S402-S403.

[0079] S402: Extract real-time data for each evaluation indicator from the real-time data stream; wherein, different types of data sources correspond to different evaluation indicators, and each data source corresponds to at least one evaluation indicator.

[0080] When the primary data source is the online customer service and feedback system, the evaluation metrics include initial response time, problem resolution rate, and feedback sentiment analysis.

[0081] Online customer service and feedback systems focus on the efficiency and quality of user feedback and problem resolution. First response time specifically refers to the time it takes for the company to respond to user feedback. Problem resolution rate specifically refers to the percentage of user issues resolved on the first submission. Feedback sentiment analysis specifically refers to analyzing the emotional tendency of user feedback content to measure positive, negative, or neutral emotions.

[0082] When the primary data source is a business system, collector 300 can be database collector 301. Evaluation metrics include browsing time, transaction frequency, total transaction value, user operation results, and service evaluation. It's understandable that problems encountered by users in the business system (such as service failures or transaction failures) can lead to increased dissatisfaction; therefore, user operation results can also be used as an evaluation metric in the scoring process. Browsing time refers to the length of time a user stays in the system. Transaction frequency refers to the number of times a user completes a transaction. Total transaction value refers to the cumulative amount of transactions completed by the user. Service evaluation refers to the user's satisfaction rating for the services provided.

[0083] When the primary data source is a CRM system, the collector 300 can be a database collector 301, and the evaluation metrics include user basic information, behavioral data, and preference data.

[0084] When the primary data source is an online survey system, the data collector 300 can be a message queue data collector 302, and the evaluation metrics include survey satisfaction and Net Promoter Score (NPS). Specifically, survey satisfaction refers to the user satisfaction score obtained through the survey. NPS refers to the likelihood that a user will recommend a service or product to others.

[0085] When the primary data source is an external data source, the collector 300 can be an API interface collector 304, and the evaluation metrics include the industry average service satisfaction and the industry average response time.

[0086] It is understandable that the evaluation indicators can be adjusted based on the actual situation, and are not limited to the aforementioned content.

[0087] S403: The real-time data is weighted and summed according to the preset weights of each evaluation indicator to obtain the real-time score.

[0088] The scores for each data source are calculated as follows:

[0089] (1) Business System Score:

[0090] Business system score = a1·Browsing time + a2·Business processing frequency + a3·Total business value + a4·Service evaluation.

[0091] (2) Online customer service and feedback system score:

[0092] Online customer service and feedback system score = b1·initial response time + b2·problem resolution rate + b3·feedback sentiment analysis.

[0093] (3) Online survey system score:

[0094] Online survey system score = c1·Survey satisfaction + c2·Net Promoter Score

[0095] (4) External data source score:

[0096] External data source score = d1·industry average service satisfaction + d2·industry average response time.

[0097] (5) CRM system score:

[0098] CRM system score = e1 · User basic information score.

[0099] It can also be expressed as: CRM system score = e1·user basic information score + e2·behavioral data + e3·preference data.

[0100] Among them, a1, a2, a3, a4, b1, b2, b3, c1, c2, d1, d2, e1, e2, and e3 are the weights of each evaluation indicator, which can be set according to the degree of influence of each evaluation indicator on the user experience. This application embodiment does not make specific limitations on this.

[0101] Furthermore, the data stream processor 400 may also include a speech-to-text processor 402 and an analysis and classification processor 403, and the collector 300 may also include a text collector 3031 and a speech collector 3032.

[0102] The text collector 3031 is specifically configured to perform the following steps S501: in response to the online customer service and feedback system generating a real-time text data stream, collect the real-time text data stream, and input it into the analysis and classification processor 403.

[0103] The voice acquisition device 3032 is specifically configured to perform the following steps S502: in response to the online customer service and feedback system generating real-time voice data, acquiring the real-time voice data stream and inputting it to the speech-to-text processor 402.

[0104] It is understood that online customer service and feedback systems may include text feedback information and voice feedback information. In this application embodiment, different processors 400 and processing methods may be configured based on different types of feedback information.

[0105] The speech-to-text processor 402 is specifically configured to perform the following steps S503: converting the real-time speech data stream into text data and inputting the text data into the analysis and classification processor 403.

[0106] The analysis and classification processor 403 is specifically configured to perform the following steps S504: classify the real-time text data stream and / or text data to obtain classification results, and input the classification results to the score calculation processor 401 so that the score calculation processor 401 calculates a real-time score based on the classification results; the classification results include classification labels for the real-time text data stream and the text data, the classification labels include business classification labels, tendency analysis labels, and / or keyword labels, and the tendency analysis labels include positive labels, neutral labels, negative labels, and severely negative labels. In this way, the real-time data stream can be converted into structured data to increase the accuracy of the scoring.

[0107] As can be seen from the above, the embodiments of this application can achieve the following technical effects:

[0108] Data integration via real-time data streaming: This embodiment of the application achieves higher real-time performance by employing real-time data streaming technology. Compared to traditional batch processing methods, data can be collected, processed, and synchronized as it is generated, ensuring data immediacy and consistency. This real-time processing capability improves the system's response speed to dynamically changing data, meeting the high requirements of modern enterprises for real-time data processing.

[0109] Data integration tasks serve as the basic integration unit: In this embodiment, data integration tasks are used as the basic integration unit, with each task containing specific data acquisition, processing, and output processes. Data flow specifications are used within each task to define data fields and formats, ensuring data consistency and readability throughout the integration process. Tasks can run independently, exhibiting high flexibility and scalability to adapt to data integration needs in different scenarios.

[0110] Easy-to-use Web-based Dataflow Task Designer: This application provides an intuitive and easy-to-use web-based dataflow task designer. The designer features a graphical interface, allowing users to easily create, manage, and optimize data integration tasks through drag-and-drop operations. The designer includes a toolbox, properties window, workspace, and menu bar, supporting the addition and configuration of nodes, the definition of connections, and the setting of execution conditions. Users can quickly get started, significantly reducing the design complexity of data integration tasks and improving development efficiency.

[0111] Real-time Customer Feedback Integration: A system that integrates customer feedback data from multiple channels, including customer feedback systems, social media, online reviews, and customer service records, is instantly consolidated. This ensures that businesses can understand customer needs, opinions, and dissatisfaction immediately, allowing for rapid adjustments to service strategies and improved customer satisfaction.

[0112] Dynamic Customer Satisfaction Monitoring: Leveraging the real-time data processing capabilities of an integrated data system, this feature dynamically monitors changes in customer satisfaction, generating real-time satisfaction scores and trend analyses. Businesses can use these dynamic indicators to quickly identify service issues and strengths, enabling timely improvements or optimizations to maintain a competitive edge.

[0113] Personalized Customer Relationship Management: By analyzing real-time data streams, the data integration system provided in this application embodiment can identify the preferences and needs of different customer groups, helping enterprises achieve personalized customer relationship management (CRM). For example, it can provide customized services for high-value customers or take retention measures in advance for potential churned customers to improve customer loyalty.

[0114] Real-time early warning and response mechanism: Establish a real-time early warning mechanism for customer satisfaction. When satisfaction indicators fluctuate abnormally, an early warning will be automatically triggered to notify relevant departments to take countermeasures. This helps companies take swift action before problems escalate and have a greater impact on the customer base, reducing customer churn.

[0115] Proprietary process for improving service quality: This application provides a customer satisfaction improvement process and strategy that can enhance a company's service standards and competitiveness.

[0116] Figure 4 This is a flowchart illustrating a method for data integration based on real-time data streams, provided in an embodiment of this application.

[0117] like Figure 4 As shown, the method for data integration based on real-time data streams provided in this application includes the following steps S601-S603.

[0118] S601: In response to the generation of a real-time data stream from a first data source, collect the real-time data stream; wherein, the first data source is one of multiple data sources, and the real-time data stream is generated based on user operations.

[0119] S602: Calculate the real-time score of the first data source based on the real-time data of various evaluation indicators extracted from the real-time data stream; and obtain the historical score calculated from the historical data stream of the second data source, and calculate the real-time user experience evaluation score based on the real-time score and the historical score. The second data source is a data source other than the first data source among multiple data streams, including at least one of the business system, CRM system, online survey system, online customer service and feedback system, and external data source.

[0120] S603: Output the user experience evaluation score in different formats depending on the output location, including database, file system or FTP server.

[0121] In some implementations, the following steps S604-S606 are also included.

[0122] S604: Output the user experience evaluation score to the historical record database to update the historical score corresponding to the first data source.

[0123] S605: Determine whether the user experience evaluation score is lower than the preset threshold.

[0124] S606: If the user experience evaluation score is lower than the preset threshold, the user experience evaluation score will be output to the dissatisfied user warning database.

[0125] Step S602 may include the following step S6021: Calculate the user experience evaluation score according to the first formula.

[0126] The first formula is: User Experience Evaluation Score = w1·Business System Score + w2·Online Customer Service and Feedback System Score + w3·Online Survey System Score + w4·External Data Source Score + w5·CRM System Score, where w1, w2, w3, w4 and w5 are the score weights.

[0127] In some implementations, step S602 may also include the following steps S6022-S6023.

[0128] S6022: Extract real-time data for various evaluation indicators from the real-time data stream; wherein, different types of data sources correspond to different evaluation indicators, and each data source corresponds to at least one evaluation indicator;

[0129] S6023: The real-time data is weighted and summed according to the preset weights of each evaluation indicator to obtain the real-time score.

[0130] In some implementations, step S602 may also include the following steps S6024-S6027.

[0131] S6024: In response to the real-time text data stream generated by the online customer service and feedback system, collect the real-time text data stream and input it into the analysis and classification processor;

[0132] S6025: Responds to the real-time voice data generated by the online customer service and feedback system, collects the real-time voice data stream, and inputs it to the speech-to-text processor;

[0133] S6026: Converts real-time voice data stream into text data and inputs the text data into the analysis and classification processor;

[0134] S6027: Classify and process the real-time text data stream and / or text data to obtain classification processing results, and input the classification processing results to the score calculation processor so that the score calculation processor can calculate the real-time score based on the classification processing results; the classification processing results include classification labels for the real-time text data stream and text data, and the classification labels include business classification labels, tendency analysis labels and / or keyword labels, and the tendency analysis labels include positive labels, neutral labels, negative labels and severely negative labels.

[0135] In some implementations, when the primary data source is an online customer service and feedback system, the evaluation metrics include initial response time, problem resolution rate, and feedback sentiment analysis.

[0136] When the primary data source is a business system, the collector is a database collector, and the evaluation metrics include browsing time, business processing frequency, total business value, user operation results, and service evaluation.

[0137] When the primary data source is a CRM system, the collector is a database collector, and the evaluation metrics include basic user information, behavioral data, and preference data.

[0138] When the primary data source is an online survey system, the data collector is a message queue data collector, and the evaluation metrics include survey satisfaction and net promoter score.

[0139] When the primary data source is an external data source, the collector is an API interface collector, and the evaluation metrics include the industry average service satisfaction and the industry average response time.

[0140] Figure 5 This is a flowchart illustrating the data flow integration method provided in an embodiment of this application.

[0141] like Figure 5 As shown, the data stream integration method provided in this application embodiment may include the following steps S701-S704.

[0142] S701: Creates a task template in response to user interaction.

[0143] S702: Add multiple task nodes to the task template and establish the connection relationship between each task node. The task node includes at least a collector node, a processor node, and an output node.

[0144] S703: Configure the task data flow specification for the task template, whereby the task data flow specification defines the field format of the data flow;

[0145] S704: Build a user analysis model based on task templates. The user analysis model is used to run on the aforementioned system that integrates data based on real-time data streams.

[0146] In some implementations, the following steps may also be included: S705: responding to the user analysis model to create a task instance based on the task template, and managing the task instance based on the task template and task data flow specifications.

[0147] In a specific implementation, the present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, it may include some or all of the steps of the method for data integration based on real-time data streams and / or the data stream integration method provided by the present invention. The storage medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0148] It is readily understood that, based on the several embodiments provided in this application, those skilled in the art can combine, split, or reorganize the embodiments of this application to obtain other embodiments, none of which exceed the protection scope of this application.

[0149] The above detailed embodiments further illustrate the purpose, technical solution, and beneficial effects of the embodiments of this application. It should be understood that the above are merely specific embodiments of the embodiments of this application and are not intended to limit the protection scope of the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solutions of the embodiments of this application should be included within the protection scope of the embodiments of this application.

Claims

1. A system for data integration based on real-time data streams, characterized in that, include: A data collector is configured to collect a real-time data stream in response to a first data source generating the real-time data stream; wherein, the data collector includes multiple types, with different types of data collectors corresponding to different types of data sources, the first data source being one of multiple data sources, and the real-time data stream being generated based on user operations; A data stream processor is configured to calculate a real-time score for the first data source based on real-time data of various evaluation metrics extracted from the real-time data stream; and to obtain historical scores calculated from historical data streams for the second data source, and to perform a weighted summation of the real-time score corresponding to the first data source and the historical scores corresponding to each of the second data sources to obtain a user experience evaluation score, wherein the second data source is any data source other than the first data source among the multiple data streams; the data source includes business systems, CRM systems, online survey systems, online customer service and feedback systems, and external data sources; The data stream processor includes a score calculation processor, which is specifically configured as follows: The real-time data for each of the evaluation metrics is extracted from the real-time data stream; wherein, different types of data sources correspond to different evaluation metrics, and each data source corresponds to at least one evaluation metric; The real-time data is weighted and summed according to the preset weights of each evaluation indicator to obtain the real-time score; A data stream outputter is configured to output the user experience evaluation score in different output formats depending on the output location, including a database, file system, or FTP server. The user experience evaluation score is output to the historical record database to update the historical score corresponding to the first data source.

2. The system for data integration based on real-time data streams according to claim 1, characterized in that, The data stream output device is specifically configured as follows: Determine whether the user experience evaluation score is lower than a preset threshold; If the user experience evaluation score is lower than the preset threshold, the user experience evaluation score will be output to the dissatisfied user warning database.

3. The system for data integration based on real-time data streams according to claim 1, characterized in that, The score calculation processor is specifically configured to calculate the user experience evaluation score according to the first formula; The first formula is: User experience evaluation score = w1·Business system score + w2·Online customer service and feedback system score + w3·Online survey system score + w4·External data source score + w5·CRM system score, where w1, w2, w3, w4 and w5 are the score weights.

4. The system for data integration based on real-time data streams according to claim 1, characterized in that, The data stream processor includes a speech-to-text processor and an analysis and classification processor, and the collector includes a text collector and a speech collector; The text collector is configured to: in response to the online customer service and feedback system generating a real-time text data stream, collect the real-time text data stream, and input it into the analysis and classification processor; The voice collector is configured to: in response to the online customer service and feedback system generating real-time voice data, collect the real-time voice data stream, and input it to the voice-to-text processor; The speech-to-text processor is configured to convert the real-time speech data stream into text data and input the text data into the analysis and classification processor. The analysis and classification processor is specifically configured to: classify the real-time text data stream and / or the text data to obtain a classification result, and input the classification result to the score calculation processor so that the score calculation processor calculates the real-time score based on the classification result; The classification processing result includes the real-time text data stream and the classification labels of the text data. The classification labels include business classification labels, tendency analysis labels and / or keyword labels. The tendency analysis labels include positive labels, neutral labels, negative labels and severely negative labels.

5. The system for data integration based on real-time data streams according to claim 1, characterized in that, When the first data source is the online customer service and feedback system, the evaluation indicators include first response time, problem resolution rate, and feedback sentiment analysis. When the first data source is the business system, the collector is a database collector, and the evaluation indicators include browsing time, business processing frequency, total business value, user operation results, and service evaluation. When the first data source is a CRM system, the collector is the database collector, and the evaluation indicators include user basic information, behavioral data, and preference data. When the first data source is the online survey system, the collector is a message queue collector, and the evaluation indicators include survey satisfaction and net promoter score. When the first data source is the external data source, the collector is an API interface collector, and the evaluation indicators include the industry average service satisfaction and the industry average response time.

6. A data flow integration platform, characterized in that, Includes a dataflow task designer, which is configured to: Create task templates in response to user interactions; Multiple task nodes are added to the task template, and the connection relationship between each task node is established. The task node includes at least a collector node, a processor node, and an output node. Configure a task data flow specification for the task template, wherein the task data flow specification defines the field format of the data flow; A user analysis model is constructed based on the task template, and the user analysis model is used to run on the system for data integration based on real-time data streams as described in any one of claims 1-5.

7. The data stream integration platform according to claim 6, characterized in that, It also includes a task runtime controller, which is configured to: In response to the user analysis model, a task instance is created based on the task template, and the task instance is managed based on the task template and the task data flow specification.

8. A method for data integration based on real-time data streams, characterized in that, include: In response to the generation of a real-time data stream from a first data source, the real-time data stream is collected; wherein the first data source is one of a plurality of data sources, and the real-time data stream is generated based on user operations; The real-time score of the first data source is calculated based on the real-time data of various evaluation indicators extracted from the real-time data stream; and the historical score calculated for the historical data stream of the second data source is obtained, and the real-time score corresponding to the first data source is weighted and summed with the historical scores corresponding to each of the second data sources to obtain the user experience evaluation score. The second data source is the other data sources in the multiple data streams besides the first data source. The data sources include business systems, CRM systems, online survey systems, online customer service and feedback systems, and external data sources. The calculation of the real-time score of the first data source based on the real-time data of various evaluation indicators extracted from the real-time data stream includes: The real-time data for each of the evaluation metrics is extracted from the real-time data stream; wherein, different types of data sources correspond to different evaluation metrics, and each data source corresponds to at least one evaluation metric; The real-time data is weighted and summed according to the preset weights of each evaluation indicator to obtain the real-time score; The user experience evaluation score will be output in different formats depending on the output location, which may include a database, file system, or FTP server. The user experience evaluation score is output to the historical record database to update the historical score corresponding to the first data source.

9. A data stream integration method, characterized in that, include: Create task templates in response to user interactions; Multiple task nodes are added to the task template, and the connection relationship between each task node is established. The task node includes at least a collector node, a processor node, and an output node. Configure a task data flow specification for the task template, wherein the task data flow specification defines the field format of the data flow; A user analysis model is constructed based on the task template, and the user analysis model is used to run on the system for data integration based on real-time data streams as described in any one of claims 1-5.

Images