A data processing method and device, computer equipment and a storage medium

Abstract

The application discloses a data processing method and device, computer equipment and a storage medium, belongs to the technical field of big data, and is applied to an insurance financial intelligent risk control and claim analysis system or an auxiliary clinical diagnosis and treatment and health evaluation system. Through unified decomposition and parallel task and flow pushing of the back end, the application avoids the request queuing and page rendering blocking problem caused by the limitation of the same source concurrent connection number of the browser, reduces the front-end resource consumption; and adopts the mechanism of parallel asynchronous task execution and flow aggregation pushing, the data slice which is processed first can be packaged and pushed to the front end in time, the flow response advantage of text generation task is fully played, the first byte response time is effectively shortened, the user can see the analysis result gradually, and the interaction experience is greatly improved. Through the non-blocking I / O and event loop mechanism, the server connection processing capacity is improved, and the operation performance and user experience of the multi-modal intelligent analysis system are improved as a whole.

Description

Technical Field

[0001] This application belongs to the field of big data technology, specifically relating to a data processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] A multimodal intelligent analysis system refers to an intelligent analysis system capable of simultaneously processing, fusing, and outputting multiple types of data (i.e., "modals"). Here, "modals" typically include different forms of information such as text, structured data, charts, audio, images, and video. This type of system uses a unified analysis and interaction framework to collaboratively process data from different sources and of different types, and outputs the analysis results to the user in diverse formats.

[0003] For example, in financial insurance scenarios, multimodal intelligent analysis systems can be applied to intelligent risk control and claims analysis. For instance, when an insurance reviewer inputs "analyze this customer's auto insurance claims risk," the system simultaneously invokes multiple analysis modules: on one hand, a large language model generates textual analysis conclusions about the customer's historical behavior, risk characteristics, and abnormal patterns; on the other hand, a data analysis engine queries structured indicator data in real time, such as the customer's historical insurance records, accident frequency, and payout amounts; simultaneously, a charting engine generates visualizations such as risk trend charts, regional accident distribution maps, and payout change curves. The system ultimately outputs text, data, and charts to a unified interactive interface, helping reviewers quickly complete risk assessments and decisions.

[0004] For example, in healthcare settings, multimodal intelligent analysis systems can assist in clinical diagnosis and health assessment. For instance, after a doctor inputs "analyze recent changes in the patient's condition and potential risks," the system can simultaneously process various data, including electronic medical records, laboratory indicators, and medical images. Specifically, the AI ​​model generates text containing condition analysis and treatment suggestions, the data analysis module calculates trends for monitoring indicators such as blood pressure, blood sugar, and heart rate, and the charting module generates curves showing changes in the patient's health and warning charts for abnormal indicators. Through the unified analysis and display of different modalities of medical data, doctors can more intuitively grasp changes in the patient's condition, improving diagnostic efficiency and accuracy.

[0005] Existing multimodal intelligent analysis systems typically employ either a multi-interface independent request mode or a BFF aggregation mode to achieve combined capabilities such as text generation, indicator querying, and chart display. In the multi-interface independent request mode, the front-end needs to initiate multiple concurrent HTTP requests simultaneously and handle data orchestration and state management independently. This is susceptible to browser limitations on the number of concurrent connections from the same origin. When multiple modules on a page load in parallel, requests exceeding the limit are queued, forcing tasks that could have been executed in parallel to be serialized, leading to page rendering blockage, slow response times, and interactive stuttering. Simultaneously, the front-end must handle complex request scheduling, exception retries, and data synchronization logic, significantly increasing system complexity and resource consumption.

[0006] On the other hand, existing BFF aggregation models typically employ a serial aggregation mechanism, requiring the backend to wait for all downstream services to complete before returning results. This results in the overall system response time being constrained by the slowest service. For example, real-time text generation results may require waiting for time-consuming data queries or chart calculations to complete before outputting, preventing the advantages of streaming response from being realized, leading to long first-byte response times and a poor user experience.

[0007] Therefore, how to reduce the complexity of front-end implementation, avoid request queuing and blocking, and allow completed service results to be output in advance to shorten the first byte response time and improve the user interaction experience while aggregating multimodal data has become an urgent problem to be solved in the development of current multimodal intelligent analysis systems. Summary of the Invention

[0008] This invention provides a data processing method, apparatus, computer equipment, and medium for artificial intelligence, which reduces the complexity of front-end implementation, avoids request queuing and blocking, allows completed service results to be output in advance, shortens the first byte response time, and improves the user interaction experience.

[0009] Firstly, a data processing method is provided, including: It receives complex intelligent analysis requests sent by clients and establishes a long-term connection channel between the server and the client based on a preset event loop mechanism and a non-blocking I / O communication model. The complex intelligent analysis request is parsed and decomposed into multiple logically isolated asynchronous task flows. The asynchronous task flows include at least a text generation task flow, an indicator data query task flow, and a chart configuration generation task flow. Based on a reactive asynchronous framework, each asynchronous task flow is executed in parallel. A unified streaming aggregator is used to asynchronously multiplex and monitor the data generated during the execution of each asynchronous task stream; When a data slice of any asynchronous task stream is detected, the data slice is encapsulated according to the corresponding data type to obtain a data message, and the encapsulated data slice is written into the transmission queue for immediate forwarding. Based on the server-sending event protocol, data messages containing event type identifiers are pushed from the transmission queue to the client through a long connection channel and rendered to the corresponding data display component.

[0010] Secondly, a data processing apparatus is provided, comprising: The connection channel module is used to receive complex intelligent analysis requests sent by the client and establish a long connection channel between the server and the client based on a preset event loop mechanism and a non-blocking I / O communication model. The request decomposition module is used to parse complex intelligent analysis requests and decompose them into multiple logically isolated asynchronous task flows. The asynchronous task flows include at least a text generation task flow, an indicator data query task flow, and a chart configuration generation task flow. Based on a reactive asynchronous framework, the asynchronous task flows are executed in parallel. The asynchronous listening module is used to asynchronously multiplex and listen to the data generated during the execution of each asynchronous task stream through a unified streaming aggregator; The message forwarding module is used to encapsulate the data slice according to the corresponding data type when any asynchronous task stream data slice is detected, obtain the data message, and write the encapsulated data slice into the transmission queue for immediate forwarding. The message display module is used to push data messages containing event type identifiers from the transmission queue to the client through a long connection channel based on the server sending event protocol, and render them to the corresponding data display component.

[0011] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described data processing method.

[0012] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described data processing method.

[0013] This application discloses a data processing method, apparatus, computer equipment, and storage medium, belonging to the field of big data technology, and applied to intelligent risk control and claims analysis systems for insurance and finance or auxiliary clinical diagnosis and health assessment systems. This application unifies the decomposition of parallel tasks and streaming pushes on the backend, eliminating the need for the frontend to initiate multiple concurrent requests. This avoids request queuing and page rendering blocking issues caused by browser same-origin concurrent connection limits, and also eliminates the need for complex request scheduling, data orchestration, and exception handling logic on the frontend, significantly reducing the complexity of frontend development and minimizing frontend resource consumption. Furthermore, by employing a mechanism of parallel asynchronous task execution plus streaming aggregation push, the backend does not need to wait for all downstream services to complete before returning results. Data slices that are processed first can be immediately packaged and pushed to the frontend, fully leveraging the streaming response advantages of tasks such as text generation, effectively shortening the first byte response time, allowing users to gradually see the analysis results, and significantly improving the interactive experience. This application improves server connection processing capabilities through non-blocking I / O and event loop mechanisms, enabling support for more concurrent client requests, and overall improving the performance and user experience of the multimodal intelligent analysis system. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram of an application environment for a data processing method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a data processing method according to an embodiment of the present invention; Figure 3 yes Figure 1 A flowchart illustrating a specific implementation of step S204; Figure 4 This is a schematic diagram of the structure of a data processing device in one embodiment of the present invention; Figure 5 yes Figure 4 A schematic diagram of a specific embodiment of the message forwarding module 404; Figure 6 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 7 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] The data processing method provided in this embodiment of the invention can be applied to, for example, Figure 1 In this application environment, the client communicates with the server via a network. The server's operations include receiving complex intelligent analysis requests from the client, establishing a long-term connection with the client based on a pre-defined event loop mechanism and a non-blocking I / O communication model, parsing and decomposing the received complex intelligent analysis request into multiple logically isolated asynchronous task flows, initiating parallel execution of each asynchronous task flow, and simultaneously monitoring the asynchronous multiplexing of data generated during the execution of each asynchronous task flow through a unified streaming aggregator. Once a new data slice is generated by an asynchronous task flow, the data slice is immediately encapsulated according to its corresponding data type to obtain a data message, written to a transmission queue for immediate forwarding, and finally, based on the server-side event protocol, pushing the data message carrying the event type identifier from the transmission queue to the client through a pre-established long-term connection channel. The client then renders the corresponding data into the corresponding data display component, completing the gradual display of the analysis results.

[0018] The client can perform the following operations: parse the event type corresponding to the data message pushed by the server, which contains an event type identifier; distribute the content of the data message to the corresponding data display component that matches the event type according to the parsed event type; splice the data slices in the order of receipt and store them in the cache of the corresponding display component, and gradually render and output incremental analysis results to realize the streaming progressive display of multimodal analysis results; when the end marker of the corresponding asynchronous task stream is detected, mark the current task stream data transmission as complete, and complete the final rendering and display of all analysis results of that modality; if the sending buffer of the long connection channel is full and cannot continue to write data, cache the remaining data to be sent and re-register the write event listener, and continue to execute data writing when the target long connection channel is writable again, until all data messages are received and displayed. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0019] Please see Figure 2 As shown, Figure 2A flowchart illustrating a data processing method provided in an embodiment of the present invention includes the following steps: S201 receives the composite intelligent analysis request sent by the client and establishes a long connection channel between the server and the client based on the preset event loop mechanism and non-blocking I / O communication model. Specifically, by constructing a network communication foundation based on EventLoop and Non-blocking I / O (NIO), and utilizing operating system multiplexing technology, a small number of fixed worker threads are used to listen for and process the ready events of massive connections. When there is no data transmission on the link, the threads are not blocked and suspended, and the thread resources are immediately released to process the I / O events of other connections. This eliminates the overhead of thread context switching and memory stacking in high-concurrency scenarios, and enables a single node to efficiently carry massive concurrent streaming connections.

[0020] EventLoop is an event loop mechanism that continuously polls for ready I / O events, distributing processable read and write events to corresponding callback logic for execution. This eliminates the need to allocate a separate persistent thread for each connection, thus achieving efficient reuse and scheduling of connection resources. Specifically, EventLoop utilizes multiplexing interfaces such as epoll to listen for read and write events of all connections in the same waiting queue. Processing logic is triggered only when the corresponding event for a connection is ready, avoiding meaningless thread idling and blocking. In this scenario, once a long-lived connection is established between the server and client, all read and write events of this long-lived connection are registered in the listening queue of the corresponding EventLoop. A fixed number of worker threads cyclically process all ready events, achieving efficient scheduling of connection resources in high-concurrency scenarios.

[0021] Non-blocking I / O communication model (NIO) is an I / O processing model that does not block the calling thread. When a read or write operation is called, if there is no data to read or write on the current link, a status flag will be returned directly instead of continuously blocking and suspending the calling thread. This allows the thread to immediately process read and write requests from other ready links, avoiding the long-term occupation of thread resources by idle connections and improving the overall concurrent connection processing capability of a single node.

[0022] Long-lived connection channels between servers and clients refer to persistent communication channels that maintain the connection throughout the entire interaction cycle between the client and the server after it is established. These channels support the server in actively pushing data to the client. In this solution, the channel is implemented based on the SSE (Server-Sent Events) protocol and is used to carry out hybrid streaming push of multimodal data slices. This eliminates the need for the client to repeatedly initiate connection requests, reducing the additional overhead caused by repeatedly establishing connections.

[0023] S202 parses the complex intelligent analysis request and breaks it down into multiple logically isolated asynchronous task flows. The asynchronous task flows include at least a text generation task flow, an indicator data query task flow, and a chart configuration generation task flow. Based on a reactive asynchronous framework, each asynchronous task flow is executed in parallel. Specifically, for single, complex intelligent analysis query commands, the backend controller breaks them down into three logically isolated, independent asynchronous task flows: a text generation task flow that calls the large language model inference cluster to generate analysis content; a metric data query task flow that calls the data platform to query corresponding business metrics; and a chart configuration generation task flow that calls the business intelligence engine to generate visualization configurations. Based on a reactive asynchronous framework, these heterogeneous data flows are allocated to their respective coroutines or thread pools for independent parallel execution. Each task flow is independent and does not block others, decoupling the processing sequence of different modalities of data. This mechanism eliminates the problem of a single slow task blocking the overall response process, avoiding unnecessary waiting delays.

[0024] Asynchronous task flows refer to task units that can execute independently without synchronously waiting for other tasks to complete. Each task has its own execution context and lifecycle, and they do not block each other during execution, thus making full use of idle processor resources and improving overall processing efficiency. In this solution, different asynchronous task flows can produce data according to their own processing speed, without being blocked by other tasks that have not yet completed.

[0025] The text generation task flow, indicator data query task flow, and chart configuration generation task flow are respectively the text flow of the LLM inference cluster, the indicator data flow of calling the data platform, and the chart configuration flow of calling the BI engine. Each task flow corresponds to the generation logic of different modal analysis results, relies on different downstream services to complete the calculation, and there is no dependency relationship between them in terms of execution order, thus satisfying the conditions for parallel execution.

[0026] Taking the intelligent risk control and claims analysis system for insurance and finance as an example, when a user inputs a natural language analysis request, "Help me analyze the high-incidence scenarios of accident insurance fraud claims in the past three months and generate trend visualization charts," the backend controller will directly break down the request into a text generation task flow that calls LLM to generate analysis text, an indicator data query task flow that queries the claim data indicators of various scenarios of accident insurance in the past three months, and a chart configuration generation task flow that generates the corresponding trend chart configuration. The three tasks will start and execute simultaneously in their respective resource pools, without waiting for one task to be completed before executing the next one.

[0027] Taking the assisted clinical diagnosis and health assessment system as an example, when a user inputs a natural language analysis request, "Help me analyze the risk of disease and provide subsequent monitoring suggestions based on the patient's blood glucose test data over the past six months," the backend controller will also break down the request into three tasks: a text generation task flow that calls the large model to generate analysis conclusions and suggestions, an indicator data query task flow that retrieves the patient's historical test indicator data, and a chart configuration generation task flow that generates a visualization configuration of blood glucose change trends. These three tasks are started and executed in parallel without blocking or waiting for each other.

[0028] S203 uses a unified streaming aggregator to asynchronously multiplex and monitor the data generated during the execution of each asynchronous task stream; Specifically, the unified streaming aggregator, acting as an intermediate bridge layer between the parallel asynchronous task stream and the front-end transmission channel, will simultaneously monitor the data output status of all parallel asynchronous task streams. Through the asynchronous multiplexing mechanism, it will simultaneously capture the ready data events of each asynchronous stream, and will not ignore the data slices produced by other streams because one stream is processing data.

[0029] In specific embodiments of this application, the unified streaming aggregator can employ merging operators provided by reactive programming frameworks to aggregate events from multiple asynchronous data sources, ensuring that every data slice produced by each data source is captured in a timely manner, without any issues of event loss or delayed capture. For example, when a text generation task stream gradually produces text fragments, or a metric data query task stream returns a single batch of metric results, the unified streaming aggregator can capture the corresponding data slices immediately, without missing any pushable data output.

[0030] S204: When a data slice of any asynchronous task stream is detected, the data slice is encapsulated according to the corresponding data type to obtain a data message, and the encapsulated data slice is written into the transmission queue for immediate forwarding. Specifically, once any asynchronous task stream produces a data slice, the unified streaming aggregator immediately marks the corresponding event type identifier based on the task type of that data slice, encapsulates it into a standard data message conforming to the SSE protocol, and then directly writes it to the front-end transmission queue for immediate forwarding. This eliminates the need to cache all processing results before unified output, achieving pipelined parallelism between data processing and data transmission. The entire encapsulation and forwarding process employs a "zero-copy" processing strategy, eliminating the need for additional memory copying of data slices, minimizing processing latency, and ensuring that data reaches the transmission queue as quickly as possible for push.

[0031] The SSE protocol is a protocol for servers to push data unidirectionally to clients based on the HTTP protocol. It allows servers to continuously send data to clients through persistent long connections. Compared with the traditional polling method, it reduces the overhead of repeatedly establishing connections. Moreover, it is more lightweight than WebSocket and does not require complex handshake negotiation. It is naturally suitable for the scenario of servers pushing data slices unidirectionally to the backend in this solution.

[0032] S205, based on the server sending event protocol, pushes data messages containing event type identifiers from the transmission queue to the client through a long connection channel, and renders them to the corresponding data display component.

[0033] Specifically, a single long-connection channel built on the SSE protocol continuously pushes different types of data messages encapsulated in the transmission queue to the client. The data messages carry corresponding event type identifiers. The client only needs to listen to this one long connection, parse out the event type identifier carried by each data message, and directly distribute the corresponding data to the page rendering component that matches the type to complete the rendering and display. This eliminates the need for the front end to initiate multiple concurrent requests at the same time, avoids the browser's same-origin concurrency limit, and eliminates the need for the front end to handle the scheduling and orchestration of multiple requests and exception handling, thus reducing the complexity of front-end development. Users can gradually see different types of analysis results, greatly improving the interactive experience.

[0034] Compared to traditional solutions, the data processing method in this application avoids the problems caused by the same-origin concurrent connection limit of the front-end browser by uniformly decomposing parallel tasks and streaming push on the back end. It also eliminates the need for complex request scheduling on the front end, significantly reducing the complexity of front-end development. At the same time, by leveraging the mechanism of parallel asynchronous and streaming real-time push, the data that is processed first can arrive at the front end for display in advance, effectively shortening the first byte response time and optimizing the user interaction experience. Relying on non-blocking I / O and event loop mechanisms, it also greatly improves the concurrent connection processing capacity of a single server, enabling more users to access the system at the same time. Ultimately, it improves the overall performance of the multimodal intelligent analysis system and the user experience.

[0035] Furthermore, the steps of receiving complex intelligent analysis requests sent by the client and establishing a long-term connection channel between the server and the client based on a preset event loop mechanism and a non-blocking I / O communication model specifically include: Receive composite intelligent analysis requests sent by clients, and perform connection authentication, protocol parsing, and session initialization processing on the composite intelligent analysis requests; An event listening thread is created based on a preset event loop mechanism, and network events corresponding to multiple client connections are monitored through I / O multiplexing technology. When the I / O ready event corresponding to the target client connection is detected, the data content of the composite intelligent analysis request is read using a preset non-blocking I / O communication model, and the reading result is encapsulated into a task to be processed. The tasks to be processed are distributed to the corresponding business processing thread pool for execution, and the long-term connection between the server and the client is maintained.

[0036] Specifically, when a new client connection request arrives, connection authentication and protocol parsing are first completed to verify the client's identity and confirm that the communication uses the SSE streaming push protocol. After initializing the session context, the read and write events of the connection are registered in the EventLoop's event listening queue. A fixed number of event listening threads listen to the I / O ready events of all connections through the epoll multiplexing interface. When the read event of the corresponding connection is ready, a non-blocking I / O model is used to read the complete request data at once, without blocking the listening threads due to data incompleteness. After reading, the encapsulated task to be processed is distributed to the business thread pool for subsequent request parsing and task decomposition operations, while keeping the current connection active and not actively releasing the connection.

[0037] By following the steps above, a large number of concurrent connections can be monitored and processed using a fixed number of threads. This avoids allocating a separate resident thread for each connection, thus preventing a large number of idle threads from occupying memory resources and the performance loss caused by frequent context switching, and effectively improving the concurrent connection carrying capacity of a single node.

[0038] Furthermore, upon detecting the I / O ready event corresponding to a target client connection, the process involves using a pre-defined non-blocking I / O communication model to read the data content of the composite intelligent analysis request and encapsulating the reading results into a task to be processed. This process specifically includes: When the event listening thread detects that the read event corresponding to the target client connection is in a ready state, it obtains the Socket channel corresponding to the target client connection. Based on a non-blocking I / O communication model, request data is read from the Socket channel and written to a preset buffer. The protocol is parsed and integrity is verified on the request data in the cache buffer to identify the request type, request parameters and session identification information corresponding to the composite intelligent analysis request; After confirming that the requested data has passed the integrity check, a task object to be processed corresponding to the composite intelligent analysis request is generated according to the preset task encapsulation rules, and a unique task identifier is assigned to the task object to be processed. Write the pending task object, which carries a unique task identifier, into the asynchronous task queue, and send the task scheduling instruction to the corresponding business processing thread pool.

[0039] Specifically, after the event listener thread captures a read-ready event, it first retrieves the Socket channel instance corresponding to the current connection. It then reads the request data from the channel using a non-blocking read interface and writes it to a preset buffer. If the read does not retrieve complete request data, it returns directly without blocking the current thread to wait for subsequent data. Reading and concatenation continue until the next read-ready event is triggered, until the request data is complete. After reading, the data in the buffer is parsed to verify the data format integrity, extract the request type, request parameters, and corresponding session identifier. Once the data is confirmed to be complete, a task object containing all information is generated and assigned a globally unique task identifier. The task is then written to the asynchronous task queue, and the business processing thread pool is notified to wake up idle threads to retrieve and execute the task. This entire process does not consume event listener thread resources, ensuring that the connection listening process is not blocked.

[0040] A Socket channel is a non-blocking network communication channel instance encapsulated based on the TCP protocol. In this solution, each active long connection corresponds to a Socket channel, and all data read and write operations are completed through this channel. The channel is configured to non-blocking mode by default, and read and write operations will directly return the execution result without suspending the calling thread.

[0041] By following the steps above, we can ensure that the non-blocking read process is executed in an orderly manner, fully acquire the requested data without blocking the event listening thread, and allow the event loop to continuously process I / O events of other connections. This fully leverages the concurrent processing advantages of the non-blocking I / O model and ensures the efficient operation of connection scheduling in high-concurrency scenarios.

[0042] Furthermore, the steps of asynchronously multiplexing and monitoring the data generated during the execution of each asynchronous task stream using a unified streaming aggregator specifically include: Establish a unified streaming aggregation channel to connect the output data of each asynchronous task stream to a preset event bus, and identify and bind each asynchronous task stream to form multiple data input sources. The event-driven mechanism is used to perform non-blocking polling and listening to multiple data input sources, and the listening status and data buffer queue of each task flow are maintained in a unified scheduling thread. The data from each asynchronous task stream monitored is timestamped and reordered to generate standardized data fragments that conform to sequential consistency. Standardized data fragments are dynamically routed and distributed according to a preset reuse strategy, and then written into a unified output channel.

[0043] Specifically, the unified streaming aggregation channel pre-allocates independent data buffer areas. Each incoming asynchronous task stream is bound to a unique stream identifier. The data produced by each stream triggers a data readiness event via the event bus, preventing data from being mixed with other streams. The unified scheduling thread uses an event-driven, non-blocking polling method, only reading data from the stream buffer when a readiness event is detected, avoiding unnecessary polling of idle streams and thus preventing the consumption of processor resources. After reading each data segment, it is rearranged according to the original timestamp of the data output, ensuring that data produced at the same time sequence enters the output channel in the correct order, preventing fragment out-of-order issues that could lead to client parsing errors. Finally, the standardized data segments are written to the unified output channel according to preset routing rules, awaiting subsequent encapsulation and forwarding. The entire listening and aggregation process is non-blocking, ensuring that it will not stop due to the incomplete readiness of a single stream, guaranteeing the orderly aggregation of multiple asynchronous data streams.

[0044] The unified streaming aggregator relies on an event-driven model for scheduling, and only performs processing operations when a data output event is triggered. It will not occupy processor computing power when there is no data, resulting in higher resource utilization compared to the synchronous polling method. It can adapt to scenarios with large fluctuations in task flow output rate and ensure the high efficiency of multi-path asynchronous data stream aggregation processing.

[0045] By following the steps above, we can ensure that the data slices produced by the multi-path asynchronous task stream are captured in a timely and accurate manner, without data loss, out-of-order processing or delayed processing issues. This allows each piece of data that is processed first to enter the subsequent forwarding and pushing process as soon as possible, giving full play to the low-latency advantage of streaming transmission and ensuring the sequential consistency and processing timeliness of multimodal data output.

[0046] Further, please refer to Figure 3 Upon detecting a data slice of any asynchronous task stream, the process involves encapsulating the data slice according to its corresponding data type to obtain a data message, and then writing the encapsulated data slice into a transmission queue for immediate forwarding. This process specifically includes: S301 monitors the data output status of each asynchronous task stream in real time through a unified streaming aggregator, and obtains the corresponding data slice content when it detects that any asynchronous task stream generates a data slice. S302, Identify the data source type corresponding to the data slice, wherein the data source type includes at least text data type, indicator data type and chart configuration data type; S303, based on the identified data source type, add corresponding event type identifiers to the data slices, and encapsulate the data slices according to the preset unified transmission protocol to generate corresponding data messages; S304, write the generated data message into the asynchronous transmission queue, and trigger the instant push of the data message to the corresponding long connection channel of the client based on the non-blocking write mechanism; After completing the data message push, S305 continues to monitor the data output status of the remaining asynchronous task flows until all asynchronous task flows have completed data processing.

[0047] Specifically, the unified streaming aggregator continuously monitors the system, executing the aforementioned identification, encapsulation, and queuing process once for each captured data slice, without waiting for all task flows to complete before batch processing. Each data slice enqueued immediately triggers a push notification, allowing the long-lived connection to promptly retrieve the data from the transmission queue and send it to the client, achieving a "output-as-forward" streaming processing effect. The unified streaming aggregator only ends its monitoring process after all asynchronous tasks have been marked as completed, pushing a task completion event to the client and closing the long-lived connection corresponding to this session. This incremental processing and forwarding mode ensures that each part of the result reaches the front end for display as quickly as possible, minimizing user waiting time, while also avoiding additional server cache space for storing complete results, resulting in lower memory usage.

[0048] Through the above steps, different types of data slices can be accurately identified and standardized and packaged, allowing each data slice to carry the corresponding type identifier into the transmission process. This facilitates quick identification and distribution by the client, avoids confusion and disorder of different types of data, and, relying on the mechanism of real-time writing and real-time push, allows data to flow without waiting for all tasks to be completed, further reducing the delay from data output to display and ensuring the immediacy of streaming push.

[0049] Furthermore, based on the identified data source type, the steps of adding corresponding event type identifiers to the data slices and encapsulating the data slices according to a preset unified transmission protocol to generate corresponding data messages specifically include: Parse the data source information corresponding to the data slice, and determine the event type identifier corresponding to the data source information according to the preset type mapping rules; Write the event type identifier into the event field of the preset message structure, and write the data slice content into the corresponding data field; Add at least one auxiliary control information from task identifier, timestamp, and data sequence number to the message structure to generate a standardized data message; Based on a pre-defined unified transmission protocol, standardized data messages are formatted and their boundaries are encapsulated to generate standard protocol messages that can be transmitted through long-connection channels.

[0050] Specifically, each data slice corresponds to a unique data source. Following preset type mapping rules, it matches the event type identifier corresponding to that data type. After filling the identifier into the `event` field, the actual business data is stored in the `data` field. Auxiliary information such as the task identifier corresponding to this request, the data generation timestamp, and the data sequence number of the current slice are then added to facilitate subsequent session matching and data verification by the client. Finally, encoding is completed according to the SSE protocol requirements, and corresponding newline delimiters are added as message boundaries, generating a standard protocol message that meets transmission requirements. This standardized message encapsulation process allows different types of data to be quickly parsed and identified by the client, ensuring the consistency of data structure during cross-node transmission and avoiding parsing errors.

[0051] The `event` field identifies the business data type corresponding to the current data block. After receiving the message, the client can directly route the data to the corresponding rendering module on the front-end page based on the value of the `event` field, without needing to perform additional data parsing and classification, thus significantly improving the efficiency of front-end data processing and rendering. Standardized message encapsulation defines a unified output format for different types of analysis results, reducing the data integration cost between the front-end and back-end while ensuring the identifiability and integrity of data during streaming transmission.

[0052] The `data` field stores the actual business analysis results corresponding to the current data slice. The data structure for different event types is predefined according to business needs. The front-end can directly parse and render data according to the corresponding structure without additional adaptation. By distinguishing between the field storing type identifiers and business data, the message structure hierarchy is clear, and the parsing logic is simple and direct. This reduces the complexity of front-end and back-end integration and improves the efficiency of message parsing and processing. Through the above steps, standardized and complete protocol messages can be generated, meeting the format requirements of SSE streaming transmission and enabling clients to quickly complete type identification and data parsing. This ensures identifiability and structural consistency during data transmission, avoiding data parsing errors or missing information.

[0053] By following the steps above, it can be ensured that different types of data slices can be correctly encapsulated into standardized messages that conform to the requirements of the transmission protocol, clearly distinguishing between type identifiers and business content. This simplifies the parsing and processing logic of the front-end and back-end, reduces the cost of integration and adaptation, and also ensures the integrity and identifiability of data during streaming transmission, avoiding problems such as missing information or parsing errors.

[0054] Furthermore, the steps of writing the generated data message into the asynchronous transmission queue and triggering the immediate push of the data message to the corresponding long connection channel of the client based on the non-blocking write mechanism specifically include: The generated data messages are written into the asynchronous transmission queue according to the corresponding client session identifier, and the association between the data messages and the target long connection channel is established. The event loop mechanism monitors the write event status of the target long connection channel and triggers a non-blocking write operation when the target long connection channel is detected to be in a writable state. A non-blocking I / O write mechanism is used to read the data message to be sent from the asynchronous transmission queue, write the data message to be sent into the target long connection channel, and perform real-time push to the client; If it is detected that the data message to be sent has not been written completely, the remaining data to be sent is retained and the write event listener is re-registered. When the target long connection channel is in a writable state again, the data writing continues until the data message to be sent is pushed.

[0055] Specifically, when data messages are written to the queue, they are bound to the corresponding client's long-lived connection channel based on the session identifier. The event loop continuously monitors the write-ready status of this channel. When the channel is writable, the data message in the queue is retrieved and non-blocking write is performed. If the current channel buffer is full, resulting in incomplete data writing, the thread is not blocked waiting for the buffer to become free. Instead, the remaining data is retained, a write event listener is re-registered, and the process returns. The remaining data continues to be written when the next write-ready event is triggered, until all data is pushed. Throughout this process, the event loop thread is not blocked, ensuring that I / O events for other connections can be processed normally. This non-blocking push mechanism ensures that large data messages can be pushed in complete chunks without blocking the entire event scheduling process due to temporary channel unwritability, keeping the push process running efficiently in high-concurrency scenarios. Through the above steps, the efficiency of data message push can be guaranteed in high-concurrency connection scenarios. The non-blocking write mechanism avoids thread blocking, fully utilizes thread resources to handle more connection transmission requests, and ensures the integrity of data push without data loss or push interruption, ensuring the stable and reliable operation of the streaming push process.

[0056] Through the above steps, efficient non-blocking push of data messages can be achieved in high-concurrency connection scenarios. With the help of the write event listening mechanism of the event loop, thread blocking when the channel is not writable is avoided, and thread resources are fully released to handle more connection requests. At the same time, the complete push of large data messages is guaranteed, and there will be no data loss or push interruption due to insufficient channel buffer capacity. This effectively ensures the stable and reliable operation of the streaming push process and adapts to the business needs of pushing a large number of long connections at the same time in high-concurrency scenarios.

[0057] In this embodiment, this application implements full-process read and write processing through a non-blocking I / O model, separates business calculation and event listening threads by combining an asynchronous task queue, and completes the ordered aggregation and incremental forwarding of multiple asynchronous data through a unified streaming aggregator. Finally, it achieves low-latency streaming output of multi-task parallel processing results in high-concurrency scenarios. This not only fully leverages the concurrent processing advantages of the non-blocking model, but also solves the problems of out-of-order results and high transmission latency of multiple asynchronous tasks. It can effectively improve the response efficiency of interactive data analysis in high-concurrency scenarios and optimize the user waiting experience.

[0058] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0059] In one embodiment, a data processing apparatus 400 is provided, which corresponds one-to-one with the data processing methods described in the above embodiments. For example... Figure 4 As shown, the data processing device 400 includes: The connection channel module 401 is used to receive composite intelligent analysis requests sent by the client and establish a long connection channel between the server and the client based on a preset event loop mechanism and a non-blocking I / O communication model. The request decomposition module 402 is used to parse the composite intelligent analysis request and decompose it into multiple logically isolated asynchronous task flows. The asynchronous task flows include at least a text generation task flow, an indicator data query task flow, and a chart configuration generation task flow. Based on a reactive asynchronous framework, each asynchronous task flow is executed in parallel. The asynchronous listening module 403 is used to asynchronously multiplex and listen to the data generated during the execution of each asynchronous task stream through a unified streaming aggregator; The message forwarding module 404 is used to encapsulate the data slice according to the corresponding data type when any asynchronous task stream data slice is detected, to obtain a data message, and write the encapsulated data slice into the transmission queue for immediate forwarding. The message display module 405 is used to push data messages containing event type identifiers from the transmission queue to the client through a long connection channel based on the server sending event protocol, and render them to the corresponding data display component.

[0060] Furthermore, the connection channel module 401 specifically includes: The request processing unit is used to receive composite intelligent analysis requests sent by the client, and to perform connection authentication, protocol parsing and session initialization processing on the composite intelligent analysis requests. The listener creation unit is used to create event listener threads based on a preset event loop mechanism, and listen to network events corresponding to multiple client connections through I / O multiplexing technology. The data encapsulation unit is used to read the data content of the composite intelligent analysis request using a preset non-blocking I / O communication model when the I / O ready event corresponding to the target client connection is detected, and encapsulate the reading result into a task to be processed. The task distribution unit is used to distribute tasks to be processed to the corresponding business processing thread pool for execution and maintain the long-term connection between the server and the client.

[0061] Furthermore, the data encapsulation unit specifically includes: The Socket channel subunit is used to obtain the Socket channel corresponding to the target client connection when the event listening thread detects that the read event corresponding to the target client connection is in a ready state. The cache write subunit is used to read request data from the Socket channel based on the non-blocking I / O communication model and write the read request data into the preset cache buffer. The data verification subunit is used to perform protocol parsing and integrity verification on the request data in the cache buffer to identify the request type, request parameters and session identification information corresponding to the composite intelligent analysis request; The task object subunit is used to generate a task object to be processed corresponding to the composite intelligent analysis request according to the preset task encapsulation rules after the request data has passed the integrity verification, and to assign a unique task identifier to the task object to be processed. The task writing sub-unit is used to write the pending task object carrying a unique task identifier into the asynchronous task queue and send the task scheduling instruction to the corresponding business processing thread pool.

[0062] Furthermore, the asynchronous listening module 403 specifically includes: The task flow binding unit is used to establish a unified streaming aggregation channel, which connects the output data of each asynchronous task flow to a preset event bus through the streaming aggregation channel, and identifies and binds each asynchronous task flow to form multiple data input sources. The polling and listening unit is used to perform non-blocking polling and listening to multiple data input sources based on an event-driven mechanism, and maintain the listening status and data buffer queue of each task flow in a unified scheduling thread. Standardized data units are used to perform timestamp alignment and stream reordering on the data from each asynchronous task stream being monitored, in order to generate standardized data fragments that conform to sequential consistency. The routing and distribution unit is used to dynamically route and distribute standardized data fragments according to a preset reuse strategy and write them into a unified output channel.

[0063] Further, please refer to Figure 5 The message forwarding module 404 specifically includes: The data slicing unit 501 is used to monitor the data output status of each asynchronous task stream in real time through a unified streaming aggregator, and to obtain the corresponding data slice content when it detects that any asynchronous task stream generates a data slice. Data source unit 502 is used to identify the data source type corresponding to the data slice, wherein the data source type includes at least text data type, indicator data type and chart configuration data type; The protocol encapsulation unit 503 is used to add a corresponding event type identifier to the data slice according to the identified data source type, and to encapsulate the data slice according to the preset unified transmission protocol to generate the corresponding data message. The message writing unit 504 is used to write the generated data message into the asynchronous transmission queue and trigger the instant push of the data message to the corresponding long connection channel of the client based on the non-blocking writing mechanism. The continuous monitoring unit 505 is used to continue monitoring the data output status of the remaining asynchronous task flows after the data message push is completed, until all asynchronous task flows have completed data processing.

[0064] Furthermore, the protocol encapsulation unit 503 specifically includes: The event type sub-unit is used to parse the data source information corresponding to the data slice and determine the event type identifier corresponding to the data source information according to the preset type mapping rules; The data identifier writing sub-unit is used to write the event type identifier into the event field in the preset message structure and write the data slice content into the corresponding data field; The standardized structure subunit is used to add at least one auxiliary control information from task identifier, timestamp and data sequence number to the message structure to generate a standardized structured data message. The message encapsulation subunit is used to encode and encapsulate standardized data messages according to a preset unified transmission protocol, generating standard protocol messages that can be transmitted through long connection channels.

[0065] Furthermore, the message writing unit 504 specifically includes: The data channel association subunit is used to write the generated data message into the asynchronous transmission queue according to the corresponding client session identifier, and to establish the association between the data message and the target long connection channel; The non-blocking write subunit is used to monitor the write event status of the target long connection channel by the event loop mechanism, and trigger a non-blocking write operation when the target long connection channel is detected to be in a writable state. The instant push subunit is used to read the data message to be sent from the asynchronous transmission queue using a non-blocking I / O write mechanism, write the data message to be sent into the target long connection channel, and perform instant push to the client; The interval write subunit is used to retain the remaining data to be sent and re-register the write event listener when it is detected that the data message to be sent has not been written. When the target long connection channel is in a writable state again, the data writing will continue until the data message to be sent is pushed.

[0066] In this embodiment, the data processing device of this application is used to execute the above-described data processing method. Specifically, after the client sends a composite intelligent analysis request, the connection channel module first completes connection authentication, protocol parsing, and session initialization, establishes a long connection based on an event loop and non-blocking I / O, and then encapsulates the request and distributes it to the business thread pool for processing. The request decomposition module decomposes the composite request into multiple independent asynchronous task streams, and allows multiple tasks to be executed in parallel based on a reactive asynchronous framework, giving full play to the computing power of multiple cores and improving the overall processing efficiency. The asynchronous listening module completes asynchronous listening of the output of multiple task streams through a unified streaming aggregator, and outputs ordered data after data sorting and alignment. The message forwarding module encapsulates the produced data slices according to type and pushes them to the client's long connection channel in real time through a non-blocking mechanism. Finally, the client parses the message according to the SSE protocol, routes the business data of the data field to the corresponding display component according to the event field to complete incremental rendering, and realizes the simultaneous generation and display of analysis results. This device solves the problems of out-of-order and delay in the streaming output of multiple asynchronous results in high-concurrency scenarios by combining a non-blocking I / O model with asynchronous task scheduling, and can effectively improve the response efficiency of interactive data analysis.

[0067] Specific limitations regarding the data processing device can be found in the limitations of the intelligent question-answering method described above, and will not be repeated here. Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0068] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a data processing method on the server side.

[0069] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a data processing method on the client side.

[0070] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: It receives complex intelligent analysis requests sent by clients and establishes a long-term connection channel between the server and the client based on a preset event loop mechanism and a non-blocking I / O communication model. The complex intelligent analysis request is parsed and decomposed into multiple logically isolated asynchronous task flows. The asynchronous task flows include at least a text generation task flow, an indicator data query task flow, and a chart configuration generation task flow. Based on a reactive asynchronous framework, each asynchronous task flow is executed in parallel. A unified streaming aggregator is used to asynchronously multiplex and monitor the data generated during the execution of each asynchronous task stream; When a data slice of any asynchronous task stream is detected, the data slice is encapsulated according to the corresponding data type to obtain a data message, and the encapsulated data slice is written into the transmission queue for immediate forwarding. Based on the server-sending event protocol, data messages containing event type identifiers are pushed from the transmission queue to the client through a long connection channel and rendered to the corresponding data display component.

[0071] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: It receives complex intelligent analysis requests sent by clients and establishes a long-term connection channel between the server and the client based on a preset event loop mechanism and a non-blocking I / O communication model. The complex intelligent analysis request is parsed and decomposed into multiple logically isolated asynchronous task flows. The asynchronous task flows include at least a text generation task flow, an indicator data query task flow, and a chart configuration generation task flow. Based on a reactive asynchronous framework, each asynchronous task flow is executed in parallel. A unified streaming aggregator is used to asynchronously multiplex and monitor the data generated during the execution of each asynchronous task stream; When a data slice of any asynchronous task stream is detected, the data slice is encapsulated according to the corresponding data type to obtain a data message, and the encapsulated data slice is written into the transmission queue for immediate forwarding. Based on the server-sending event protocol, data messages containing event type identifiers are pushed from the transmission queue to the client through a long connection channel and rendered to the corresponding data display component.

[0072] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0073] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0074] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0075] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0076] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A data processing method, characterized in that, include: It receives complex intelligent analysis requests sent by clients and establishes a long-term connection channel between the server and the client based on a preset event loop mechanism and a non-blocking I / O communication model. The composite intelligent analysis request is parsed and decomposed into multiple logically isolated asynchronous task flows. The asynchronous task flows include at least a text generation task flow, an indicator data query task flow, and a chart configuration generation task flow. The asynchronous task flows are executed in parallel based on a reactive asynchronous framework. A unified streaming aggregator is used to asynchronously multiplex and monitor the data generated during the execution of each asynchronous task stream; When a data slice of any of the asynchronous task streams is detected, the data slice is encapsulated according to the corresponding data type to obtain a data message, and the encapsulated data slice is written into the transmission queue for immediate forwarding. Based on the server sending event protocol, a data message containing an event type identifier is pushed from the transmission queue to the client through the long connection channel and rendered to the corresponding data display component.

2. The data processing method as described in claim 1, characterized in that, The steps of receiving the composite intelligent analysis request sent by the client and establishing a long-term connection channel between the server and the client based on a preset event loop mechanism and a non-blocking I / O communication model specifically include: Receive composite intelligent analysis requests sent by clients, and perform connection authentication, protocol parsing, and session initialization processing on the composite intelligent analysis requests; An event listening thread is created based on a preset event loop mechanism, and network events corresponding to multiple client connections are monitored through I / O multiplexing technology. When the I / O ready event corresponding to the target client connection is detected, the data content of the composite intelligent analysis request is read using a preset non-blocking I / O communication model, and the reading result is encapsulated as a task to be processed. The tasks to be processed are distributed to the corresponding business processing thread pools for execution, and the long-term connection between the server and the client is maintained.

3. The data processing method as described in claim 2, characterized in that, The step of reading the data content of the composite intelligent analysis request using a preset non-blocking I / O communication model and encapsulating the reading result into a task to be processed when the I / O ready event corresponding to the target client connection is detected specifically includes: When the event listening thread detects that the read event corresponding to the target client connection is in a ready state, it obtains the Socket channel corresponding to the target client connection. Based on the non-blocking I / O communication model, request data is read from the Socket channel and written into a preset buffer. The request data in the cache buffer is parsed and its integrity is verified to identify the request type, request parameters and session identification information corresponding to the composite intelligent analysis request; After confirming that the requested data has passed the integrity check, a task object to be processed corresponding to the composite intelligent analysis request is generated according to the preset task encapsulation rules, and a unique task identifier is assigned to the task object to be processed. Write the pending task object carrying the unique task identifier into the asynchronous task queue, and send the task scheduling instruction to the corresponding business processing thread pool.

4. The data processing method as described in claim 1, characterized in that, The step of asynchronously multiplexing and monitoring the data generated during the execution of each asynchronous task stream using a unified streaming aggregator specifically includes: A unified streaming aggregation channel is established, and the output data of each asynchronous task stream is connected to a preset event bus through the streaming aggregation channel. Each asynchronous task stream is identified and bound to form multiple data input sources. The event-driven mechanism is used to perform non-blocking polling and listening to the multiple data input sources, and the listening status and data buffer queue of each task flow are maintained in a unified scheduling thread. The data from each of the asynchronous task streams monitored are timestamped and reordered to generate standardized data fragments that conform to sequential consistency. The standardized data fragments are dynamically routed and distributed according to a preset reuse strategy, and then written into a unified output channel.

5. The data processing method as described in claim 1, characterized in that, The step of encapsulating the data slice according to the corresponding data type to obtain a data message when any asynchronous task stream data slice is detected, and writing the encapsulated data slice into a transmission queue for immediate forwarding, specifically includes: The unified streaming aggregator monitors the data output status of each asynchronous task stream in real time and obtains the corresponding data slice content when it detects that any asynchronous task stream generates a data slice. Identify the data source type corresponding to the data slice, wherein the data source type includes at least text data type, indicator data type, and chart configuration data type; Based on the identified data source type, a corresponding event type identifier is added to the data slice, and the data slice is encapsulated according to a preset unified transmission protocol to generate a corresponding data message. The generated data message is written to the asynchronous transmission queue, and the data message is triggered to be pushed to the corresponding long connection channel of the client in real time based on the non-blocking write mechanism; After the data message push is completed, continue to monitor the data output status of the remaining asynchronous task flows until all asynchronous task flows have completed data processing.

6. The data processing method as described in claim 5, characterized in that, The steps of adding a corresponding event type identifier to the data slice based on the identified data source type, and encapsulating the data slice according to a preset unified transmission protocol to generate a corresponding data message specifically include: The data source information corresponding to the data slice is parsed, and the event type identifier corresponding to the data source information is determined according to the preset type mapping rules; Write the event type identifier into the event field of the preset message structure, and write the data slice content into the corresponding data field; Add at least one auxiliary control information, including task identifier, timestamp, and data sequence number, to the message structure to generate a standardized data message; The standardized data message is format-encoded and boundary-encapsulated according to a preset unified transmission protocol to generate a standard protocol message that can be transmitted through the long connection channel.

7. The data processing method as described in claim 5, characterized in that, The step of writing the generated data message into the asynchronous transmission queue and triggering the immediate push of the data message to the corresponding long connection channel of the client based on the non-blocking write mechanism specifically includes: The generated data message is written into the asynchronous transmission queue according to the corresponding client session identifier, and the association between the data message and the target long connection channel is established. The event loop mechanism monitors the write event status corresponding to the target long connection channel, and triggers a non-blocking write operation when the target long connection channel is detected to be in a writable state. A non-blocking I / O write mechanism is used to read the data message to be sent from the asynchronous transmission queue, write the data message to be sent into the target long connection channel, and perform real-time push to the client; If it is detected that the data message to be sent has not been written, the remaining data to be sent is retained and the write event listener is re-registered. When the target long connection channel is writable again, the data writing continues until the data message to be sent is pushed.

8. A data processing apparatus, characterized in that, include: The connection channel module is used to receive complex intelligent analysis requests sent by the client and establish a long connection channel between the server and the client based on a preset event loop mechanism and a non-blocking I / O communication model. The request decomposition module is used to parse the composite intelligent analysis request and decompose it into multiple logically isolated asynchronous task flows. The asynchronous task flows include at least a text generation task flow, an indicator data query task flow, and a chart configuration generation task flow. Based on a reactive asynchronous framework, the asynchronous task flows are executed in parallel. The asynchronous listening module is used to perform asynchronous multiplexing listening on the data generated during the execution of each asynchronous task stream through a unified streaming aggregator; The message forwarding module is used to encapsulate the data slice according to the corresponding data type when any data slice of the asynchronous task stream is detected, to obtain a data message, and write the encapsulated data slice into the transmission queue for immediate forwarding; The message display module is used to push data messages containing event type identifiers from the transmission queue to the client through the long connection channel based on the server sending event protocol, and render them to the corresponding data display component.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the data processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the data processing method as described in any one of claims 1 to 7.

Images