Data processing method and device, electronic equipment and storage medium

Abstract

The invention relates to a data processing method and device, electronic equipment and a storage medium, the method is applied to a client, and the method comprises the following steps: for data currently cached in a first preset cache region, segmenting the data according to a preset boundary identifier to obtain a plurality of row data; the first preset cache region is used for continuously caching data fragments sent by the server; aiming at each row of data, analyzing the row of data; under the condition that the row data is successfully analyzed, removing the row data from the first preset cache region so as to carry out data processing on the row data; and under the condition that the analysis of the row data fails, retaining the row data in the first preset cache region so as to merge the row data with subsequent data received by the first preset cache region. Therefore, the data loss rate caused by data fragmentation can be reduced, and the arrival rate and integrity of the data stream sent to the client by the server can be improved, so that the fault-tolerant capability and continuity of data processing can be improved.

Description

Technical Field

[0001] This application relates to the field of front-end data technology, and in particular to a data processing method, apparatus, electronic device and storage medium. Background Technology

[0002] Server Push Events (SSE) is an HTTP-based technology protocol that allows servers to push data unidirectionally to clients in real time. In web applications, it is commonly used to implement scenarios requiring continuous data updates, such as real-time notifications, market data pushes, and log stream transmission. Clients receive data streams from the server by creating an EventSource connection.

[0003] In related technologies, the client-side process for handling SSE data streams is as follows: upon receiving each data fragment from the server, the client attempts to process the received fragment as an independent and complete data unit. This processing method can directly call the `JSON.parse()` method to parse the fragment. If parsing is successful, the resulting object is passed to subsequent business logic; if parsing fails, the fragment is discarded or an error is logged in the console.

[0004] However, since a logically complete JSON object may be split into multiple fragments at the transport layer during actual transmission, the client parses each data fragment as an independent unit on the fly. When a fragment happens to be an incomplete JSON, parsing will inevitably fail, resulting in the discarding of valid data. Summary of the Invention

[0005] This application provides a data processing method, apparatus, electronic device, and storage medium to solve the technical problem in the prior art where parsing fails and valid data is discarded when the client performs real-time parsing on the received data segments sent by the server as independent units.

[0006] Firstly, this application provides a data processing method applied to a client, the method comprising: For the data currently cached in the first preset cache area, the data is segmented according to a preset boundary identifier to obtain multiple rows of data; the first preset cache area is used to continuously cache data fragments sent by the server; For each row of data, the row of data is parsed; If the row data is successfully parsed, the row data is removed from the first preset cache area for data processing. If the parsing of the row data fails, the row data is retained in the first preset cache area for merging with subsequent data received in the first preset cache area.

[0007] As an optional implementation, the row data is removed from the first preset cache area to perform data processing on the row data, including: Obtain the current data processing mode, which includes an incremental processing mode; When the data processing mode is incremental processing mode, the row data is transferred from the first preset cache area to the second preset cache area. When the successfully parsed row data meets the first preset condition, the row data stored in the second preset cache area is removed based on the storage time. The second preset cache area is used to store the row data to be processed.

[0008] As an optional implementation, the row data is removed from the first preset cache area to perform data processing on the row data, including: Obtain the current data processing mode, which includes batch processing mode; When the data processing mode is batch processing mode, the row data is transferred from the first preset cache area to the second preset cache area. When the successfully parsed row data meets the second preset condition, all row data in the second preset cache area is removed. The second preset cache area is used to store the row data to be processed.

[0009] As an optional implementation, the step of retaining the row data in the first preset cache area when parsing the row data fails includes: If parsing the row data fails, determine whether the row data is the last row of data; If it is determined that the row data is the last row data, the row data will be retained in the first preset cache area; If it is determined that the row data is not the last row, the row data is deleted from the first preset cache area.

[0010] As an optional implementation, the server sends the data fragment via a network connection; the method further includes: Monitor the network connection status with the server; If the network connection is detected to be disconnected, determine the current number of reconnections; Based on the number of reconnections, determine the corresponding reconnection delay duration; Based on the reconnection delay duration, initiate a reconnection operation.

[0011] As an optional implementation, the current reconnection delay duration is determined based on the number of reconnections, including: The initial reconnection delay time is obtained by exponentiation of the number of reconnections to a preset value. The initial reconnection delay time is compared with the preset maximum reconnection delay time; The minimum value between the initial reconnection delay and the maximum reconnection delay is determined as the current reconnection delay.

[0012] As an optional implementation, the method further includes: Obtain current status data, which includes one or more of the following: data processing status, data interaction status with the server, and network connection status; The status data is stored in a preset storage medium; the storage medium has a corresponding status interface, and the status interface provides the status data to the outside world in response to a received status query request.

[0013] Secondly, this application provides a data processing apparatus for use on a client side, the apparatus comprising: The data segmentation module is used to segment the data currently cached in the first preset cache area according to a preset boundary identifier to obtain multiple rows of data; the first preset cache area is used to continuously cache data fragments sent by the server. The data parsing module is used to parse the data for each row. The data removal module is used to remove the row data from the first preset cache area when the row data is successfully parsed, so as to process the row data. The data retention module is used to retain the row data in the first preset cache area in the event that the row data parsing fails, so as to merge it with the data subsequently received in the first preset cache area.

[0014] Thirdly, this application provides an electronic device, including: a processor and a memory, wherein the processor is configured to execute a data processing program stored in the memory to implement the data processing method described in any one of the first aspects.

[0015] Fourthly, this application provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method described in any of the first aspects.

[0016] The technical solution provided in this application, by processing successfully parsed data and retaining and merging unparsed data with subsequent data, allows data incomplete due to instantaneous network fragmentation to be completed and successfully parsed in subsequent transmissions. This significantly reduces the data loss rate caused by data fragmentation and improves the final effective arrival rate and integrity of the data stream. Simultaneously, since parsing failure no longer necessarily leads to data discarding and process termination, the fault tolerance and continuity of the entire data processing process are enhanced. This reduces the data loss rate caused by data fragmentation, improves the arrival rate and integrity of the data stream sent from the server to the client, and thus enhances the fault tolerance and continuity of data processing. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0020] Figure 1 A flowchart illustrating an embodiment of a data processing method provided in this application; Figure 2 This application provides a schematic diagram illustrating data changes in a first preset buffer area. Figure 3 A flowchart illustrating an embodiment of another data processing method provided in this application; Figure 4 A flowchart illustrating another embodiment of the data processing method provided in this application; Figure 5 A flowchart illustrating another embodiment of the data processing method provided in this application; Figure 6 A block diagram illustrating an embodiment of a data processing apparatus provided in this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] The following disclosure provides numerous different embodiments or examples for implementing various structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of the invention. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0023] To address the technical problem in existing technologies where clients experience parsing failures and discarded valid data when parsing received data fragments from a server as independent units, this application provides a data processing method, apparatus, electronic device, and storage medium. This method allows for subsequent processing of successfully parsed data, while retaining and merging failed-parsing data with subsequent data. This enables data incomplete due to instantaneous network fragmentation to be completed and successfully parsed in subsequent transmissions, significantly reducing data loss caused by data fragmentation and improving the final effective arrival rate and integrity of the data stream. Simultaneously, since parsing failure no longer necessarily leads to data discarding and process termination, the fault tolerance and continuity of the entire data processing process are enhanced. This reduces data loss caused by data fragmentation, improves the arrival rate and integrity of the data stream sent from the server to the client, and ultimately enhances the fault tolerance and continuity of data processing.

[0024] The data processing method provided in this application will be further explained and described below with reference to the accompanying drawings and specific embodiments. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0025] See Figure 1 This is a flowchart illustrating an embodiment of a data processing method provided in this application. As one embodiment, Figure 1 The illustrated process can be applied to the client side. For example... Figure 1 As shown, the process may include the following steps: Step 101: For the data currently cached in the first preset cache area, the data is divided according to the preset boundary identifier to obtain multiple rows of data; the first preset cache area is used to continuously cache data fragments sent by the server.

[0026] The aforementioned first preset cache area refers to a dynamic storage area pre-allocated in the client's memory for continuously caching raw data fragments received from the server in sequence, in order to support subsequent line segmentation and parsing processing.

[0027] The aforementioned data refers to all currently cached data within the first preset cache area. Since successfully parsed data is removed for subsequent data processing, the data here may include currently unparsed data as well as data that failed to be parsed in the previous attempt.

[0028] The aforementioned boundary identifiers refer to specific characters or character sequences used in a continuous data stream to define the end position of a single complete data unit, such as the newline character (\n). Their function is to accurately divide the accumulated raw data stream into data lines that can be parsed independently.

[0029] The aforementioned row data refers to the data corresponding to each complete data unit obtained after dividing the data in the first preset buffer area. For example, the data is divided into multiple rows according to the newline character, and each row is called the first row data.

[0030] In this step, when the client decides to perform data processing, it can perform data processing on the data in the first preset cache area.

[0031] In practical applications, servers and clients can exchange data via the SSE (Server-Sent Events) protocol. During this interaction, to accelerate data transmission, the server can segment the data to be sent according to preset rules (e.g., data volume), resulting in multiple data fragments. These fragments are then sent to the client sequentially in the order they were generated.

[0032] After receiving data segments sent by the server, the client can store the data segments sequentially into a first preset buffer. The first preset buffer can then accumulate multiple data segments in the order they were received.

[0033] Based on this, when the client determines to process data, it can obtain the currently stored data in the first preset cache area and divide the data according to the preset boundary identifier to obtain multiple rows of data.

[0034] In one embodiment, the client can determine that data processing is currently underway when it determines that it has received a new data segment and stores the data segment in a first preset buffer.

[0035] In another embodiment, the client can periodically process the data in the first preset buffer. Therefore, the client can determine whether the time interval between the current time and the previous data processing time reaches a preset time interval threshold, and if the time interval reaches the aforementioned time interval threshold, determine that data processing is currently performed.

[0036] In another embodiment, the client can obtain the amount of data stored in the first preset cache area, and determine that data processing is to be performed if the amount of data is greater than or equal to a preset data amount threshold.

[0037] In one embodiment, after determining to perform data processing, the client can segment the currently cached data in the first preset cache area according to a preset boundary identifier to obtain multiple rows of data.

[0038] As an optional implementation, the boundary identifier mentioned above can be a newline character. Based on this, the client can identify the newline characters in the data and divide the data into multiple lines according to the newline characters, thus obtaining multiple rows of data.

[0039] As an example implementation, the client can use each newline character detected as the last identifier of the current line of data, and use the data following the newline character as the first data of the next line. This process can be repeated to divide the data in the first preset buffer into multiple lines of data.

[0040] Step 102: For each row of data, parse that row of data.

[0041] Step 103: If the row data is successfully parsed, remove the row data from the first preset cache area to process the row data.

[0042] Step 104: If parsing of the row data fails, the row data is retained in the first preset cache area for merging with the data subsequently received in the first preset cache area.

[0043] The following provides a unified explanation of steps 102 to 104: The above analysis refers to the process of converting a data string that conforms to a specific format into a structured data object that can be manipulated by the program. Its core action is to perform JSON decoding on the split rows of data.

[0044] In this step, after the data is divided into multiple rows, in order to determine whether each row is a complete data unit, the client can parse each row and process the row based on the parsing results.

[0045] In one embodiment, when parsing line data, the client can use a preset parsing function to process the line data. This parsing function can be a pre-defined function for parsing JSON data, such as JSON.parse(line).

[0046] Based on the above results, in one embodiment, if the client successfully parses the row data, it indicates that the row data is a complete data unit. Therefore, the row data can be removed from the first preset cache area for data processing. It is understood that the data processing described here and thereafter refers to processing the object data obtained after successfully parsing the row data.

[0047] The specific process of removing row data from the first preset cache and processing the data will be explained below, and will not be detailed here.

[0048] In another embodiment, if the client fails to parse the row data, it indicates that the row data may be an incomplete data unit or there may be other anomalies. In order to prevent the row data from affecting data processing and to prevent the invalid data from being discarded due to incomplete row data, the client can retain the row data that failed to be parsed in the first preset buffer so that the row data can be merged with the subsequently received data.

[0049] As an optional implementation, since in practical applications, a row may be incomplete only when it is the last row of data, and can be merged with subsequently received data to form complete data, the client can determine whether the row is the last row of data when the row parsing fails.

[0050] Optionally, if it is determined that the above row data is the last row data, the above row data can be retained in the first preset cache area.

[0051] Optionally, if it is determined that the above row data is not the last row data, it means that the row data has other abnormalities and cannot be successfully parsed. Therefore, in order to save storage space in the first preset cache area, the above row data can be deleted from the first preset cache area.

[0052] To facilitate understanding of the data processing method provided in this application, the following exemplary description is provided: See Figure 2This is a schematic diagram illustrating data changes in a first preset buffer area provided in an embodiment of this application. Figure 2 As shown, the data currently stored in the first preset cache area is: "{"id":1,"name":"user1"}\n{"id":2,"name":"user2"}\n{"id":2,"data":{"items":[1,2"}".

[0053] Afterwards, the data is segmented according to the boundary identifiers, resulting in three rows of data: Line1: {"id": 1, "name": "user1"}, Line2: {"id": 2, "name": "user2"}, Line3: {"id": 2, "data": {"items": [1, 2, ..., "items"}.

[0054] Analysis shows that when parsing the three rows of data, the first row (Line1) and the second row (Line2) are complete rows, so they can be successfully parsed. However, the third row (Line3) is incomplete, so it cannot be successfully parsed.

[0055] In summary, the parsing results are as follows: two object data [obj1, obj2] were successfully parsed and removed from the first preset cache for subsequent data processing; while the data that failed to be parsed is the third row of data, so the third row of data can be kept in the first preset cache.

[0056] In this scenario, when the client receives the next data segment, the next data segment can be merged with the third row of data mentioned above, thereby completing the third row of data. This avoids discarding the third row of data as invalid data, which would otherwise result in the discarding of valid data.

[0057] The technical solution provided in this application segmentes the data currently cached in a first preset buffer according to preset boundary identifiers to obtain multiple rows of data. The first preset buffer continuously caches data fragments sent by the server. For each row of data, it is parsed. If parsing is successful, the row of data is removed from the first preset buffer for further processing. If parsing fails, the row of data is retained in the first preset buffer for merging with subsequent data received from the first buffer. This technical solution, by processing successfully parsed data and retaining and merging unparsed data with subsequent data, allows data incomplete due to instantaneous network fragmentation to be completed and successfully parsed in subsequent transmissions. This significantly reduces the data loss rate caused by data fragmentation and improves the final effective arrival rate and integrity of the data stream. At the same time, since parsing failure no longer necessarily leads to data loss and process termination, the fault tolerance and continuity of the entire data processing process are also enhanced. This can reduce the data loss rate caused by data fragmentation, improve the arrival rate and integrity of the data stream sent from the server to the client, and thus improve the fault tolerance and continuity of data processing.

[0058] See Figure 3 This is a flowchart of an embodiment of another data processing method provided in this application. Figure 3 The process shown is in Figure 1 Based on the illustrated process, this section describes how to remove a row of data from the first preset cache in the event of a row data parsing failure, so that the data can be processed. For example... Figure 3 As shown, the process may include the following steps: Step 301: Obtain the current data processing mode.

[0059] The aforementioned data processing mode refers to a pre-defined data clearing mode used to clear successfully parsed row data. This data processing mode may include incremental processing mode and batch processing mode.

[0060] The above incremental processing mode refers to the process where, after the cache capacity reaches its limit, the oldest cached row of data is cleared after each row of data is successfully parsed.

[0061] The aforementioned batch processing mode refers to the process of uniformly clearing multiple rows of data that have been successfully parsed when the number of successfully parsed rows reaches a preset amount.

[0062] In this step, when clearing and processing the successfully parsed row data, in order to improve memory utilization, multiple different data processing modes can be preset, so that different data processing modes can be adopted in different application scenarios.

[0063] Based on this, when clearing successfully parsed row data and performing subsequent data processing, the client can first obtain the current data processing mode.

[0064] As an optional implementation, the aforementioned data processing mode can be preset by the user. Based on this, the client can obtain the preset data processing mode.

[0065] As an alternative implementation, the client can obtain the application scenario of the data currently stored in the first preset cache area and determine the corresponding data processing mode based on the scenario characteristics. These scenario characteristics may include, but are not limited to, the real-time nature and integrity of data application within the application scenario, as well as memory usage.

[0066] For example, for front-end interfaces that need to continuously display the latest status and respond quickly (such as real-time monitoring dashboards, stock market charts, and dynamic log displays), incremental processing mode should be configured. This mode retains only recent data through a sliding window, ensuring that the interface updates instantly and memory is manageable. On the other hand, for background tasks that focus on periodic analysis and require complete datasets for statistics or persistence (such as user behavior analysis summarized every five minutes, batch report generation, and offline data synchronization), batch processing mode should be used. This mode accumulates the complete dataset within a single period and then processes and replaces it uniformly to ensure the integrity of the data batch and processing efficiency.

[0067] Step 302: According to the data processing mode, remove the row data from the first preset buffer to process the row data.

[0068] The aforementioned second preset buffer area refers to a logical storage area used to receive and store row data that has been successfully parsed and is ready for data processing. It can store the successfully parsed row data in the form of a data set or a state array.

[0069] In this step, the client can transfer row data from the first preset cache area to the second preset cache area according to the acquired data processing mode.

[0070] In one embodiment, removing row data from a first preset cache area to process the row data includes: When the data processing mode is incremental, row data is transferred from the first preset cache to the second preset cache. When the successfully parsed row data meets the first preset condition, the row data stored in the second preset cache is removed based on the storage time. The second preset cache is used to store row data to be processed.

[0071] As an optional implementation, when the data processing mode described above is an incremental processing mode, when transferring row data from the first preset cache to the second preset cache, the oldest historical data stored in the second preset cache can be removed based on the amount of row data. This operation can automatically delete useless old data when the memory size of the second preset cache is fixed, thereby ensuring the real-time performance of data storage. In this embodiment, the first preset condition may be that the memory size of the second preset cache reaches its upper limit.

[0072] For example, suppose the data currently stored in the second preset cache is: {[1], [2], [3], [4]...

[99] ,

[100] }, and suppose the object obtained from the currently successfully parsed row data is

[101] . Then, according to the above incremental processing mode, the object data [1] can be deleted from the second preset cache first, and then the object

[101] can be stored in the latest position of the second preset cache to obtain the following latest data: {[2], [3], [4], [5]...

[100] ,

[101] }.

[0073] In another embodiment, the row data is removed from a first preset cache area to perform data processing on the row data, including: When the data processing mode is batch processing mode, row data is transferred from the first preset cache to the second preset cache. When the successfully parsed row data meets the second preset condition, all row data in the second preset cache is removed. The second preset cache is used to store row data to be processed.

[0074] As an optional implementation, in batch processing mode, if the first preset cache stores all parsed row data from the previous batch, and all row data from the next batch has been parsed, the parsed row data from the next batch is transferred from the first preset cache to the second preset cache. This allows all successfully parsed row data from the next batch to replace the historical data in the second preset cache, thus ensuring data integrity. The second preset condition can be that all row data from the next batch has been parsed.

[0075] As an exemplary implementation, after successfully parsing the row data, the total amount of the successfully parsed row data can be determined. When the total amount of the successfully parsed row data is greater than a preset data amount threshold (e.g., the amount of data corresponding to the second preset cache), all the successfully parsed row data will simultaneously replace the historical data in the second preset cache.

[0076] As another exemplary implementation, after successfully parsing the row data, it can be determined whether all row data has been parsed. If it is determined that all row data has been parsed, all successfully parsed row data can be used to replace the historical data in the second preset cache.

[0077] For example, suppose the data currently stored in the second preset buffer is: {[1], [2], [3], [4]...

[99] ,

[100] }, and further suppose the objects obtained from all the successfully parsed rows are {

[101] ,

[102] ,

[103] ,

[104] ...

[199] ,

[200] }. Then, according to the batch processing mode described above, the objects {

[101] ,

[102] ,

[103] ,

[104] ...

[199] ,

[200] } can be directly overwritten to cover all the data in the second preset buffer, resulting in the following latest data: {

[101] ,

[102] ,

[103] ,

[104] ...

[199] ,

[200] }. When the amount of the latest successfully parsed rows differs from the amount of data in the second preset buffer, all the data in the second preset buffer can be deleted first, and then all the successfully parsed rows can be written into the second preset buffer.

[0078] The technical solution provided in this application obtains the current data processing mode, which includes an incremental processing mode or a batch processing mode. Based on this mode, row data is removed from a first preset cache area for processing. This technical solution, by introducing a configurable data processing mode, achieves refined and scenario-specific adaptation of data processing logic. This mechanism decouples the parsed data aggregation strategy from a fixed, single algorithm, allowing the system to dynamically switch between incremental and batch processing modes according to runtime requirements (such as high real-time requirements or an emphasis on data integrity). The incremental processing mode limits the data volume through a sliding window, significantly optimizing memory usage in long-running scenarios and avoiding memory leaks and garbage collection pressure, making it suitable for front-end displays such as real-time dashboards. The batch processing mode ensures the atomicity and processing efficiency of data operations by accumulating and replacing the entire dataset, facilitating subsequent batch calculations or persistent storage. This improves the applicability and resource utilization efficiency of the single data stream processing framework. Simultaneously, it isolates different strategies to independent cache areas for processing, enhancing the system's modularity, maintainability, and responsiveness to different business needs.

[0079] See Figure 4 This is a flowchart of another data processing method provided in this application. Figure 4 The process shown is in Figure 1 Based on the illustrated process, this section describes how the client monitors the network status with the server and reconnects when the server sends data fragments via a network connection. For example... Figure 4 As shown, the process may include the following steps: Step 401: Monitor the network connection status with the server.

[0080] In this step, data transmission occurs between the client and server via a network connection. To ensure the stability of data transmission between the client and server, the client can monitor the network connection status with the server in real time or periodically.

[0081] In one embodiment, the client can send probe data packets to the server in real time or at regular intervals, and determine whether it receives a response from the server, as well as the round-trip delay when the response is received.

[0082] Optionally, if the above response information is not received within a preset time period, or if the round-trip delay time corresponding to the received response information is greater than a preset time threshold, the network connection is determined to be interrupted.

[0083] Optionally, if it is determined that a response message has been received and the round-trip delay corresponding to the response message is less than or equal to a preset duration threshold, the network connection is determined to be normal.

[0084] Step 402: If a network connection is detected to be lost, determine the current number of reconnections.

[0085] Step 403: Based on the above number of reconnections, determine the current reconnection delay duration.

[0086] Step 404: Initiate a reconnection operation based on the reconnection delay duration mentioned above.

[0087] The following provides a unified explanation of steps 402 to 404: The reconnection count mentioned above refers to the number of times the client initiates a network reconnection with the server after the network connection between the client and the server is lost.

[0088] The reconnection operation mentioned above refers to the network reconnection operation initiated by the client to the server when the network connection between the client and the server is lost.

[0089] The aforementioned reconnection delay refers to the interval between when the network connection between the client and the server is broken or a reconnection operation is initiated, and the next reconnection operation is initiated.

[0090] In this step, if the client detects that the network connection with the server has been lost, it can proactively initiate a network reconnection operation with the server in order to ensure data transmission between the two servers.

[0091] In practical applications, when a large number of clients simultaneously lose network connections with the server, if all clients initiate reconnection operations at fixed intervals as in existing technologies, the server may receive a large number of network connection requests in a short period of time, leading to excessive server pressure and eventual crash.

[0092] To address this, in this step, before initiating a reconnection operation with the server, the client can first determine the number of reconnections already initiated and, based on that number, determine the corresponding reconnection delay. Then, it can initiate the reconnection operation according to this delay. This step, linking the reconnection delay to the number of reconnections, prevents clients with different reconnection counts from initiating reconnection operations simultaneously, thus staggering network connection requests from different clients and reducing server load.

[0093] In one embodiment, when the client determines the current reconnection delay duration based on the number of reconnections, it can perform a preset exponentiation operation on the number of reconnections to obtain the initial reconnection delay duration.

[0094] As an optional implementation, the above number of reconnection attempts can be input into the exponentiation formula shown in equation (i) to obtain the initial reconnection delay duration: Formula (1) Among them, the above The initial reconnection delay duration is as described above. The preset baseline reconnection delay time, for example, 1 second, is as follows. This represents the number of reconnections.

[0095] Subsequently, in order to avoid excessive reconnection delay, which could lead to prolonged network interruption between the client and the server and thus data transmission interruption, the client can compare the initial reconnection delay with a preset maximum reconnection delay (e.g., 30 seconds) after determining the initial reconnection delay.

[0096] Finally, the minimum value between the initial reconnection delay and the maximum reconnection delay can be determined as the current reconnection delay.

[0097] For example, assuming the baseline reconnection delay shown in equation (1) is 1 second and the maximum delay is 30 seconds, then after the network connection between the client and the server is interrupted, the delay can increase exponentially from 1 second, with a maximum delay limit of 30 seconds. The initial reconnection is fast (1 second), which can accommodate temporary network jitter; the delay increases exponentially (2 seconds, 4 seconds, 8 seconds...), which can reduce server load; setting an upper limit (30 seconds) can ensure user experience; and the counter is reset after a successful connection, restoring fast reconnection. Compared to fixed-interval reconnection, server load is reduced by 60%, the reconnection success rate is increased by 95%, and the "swarm effect" of a large number of clients simultaneously initiating network connection requests to the server is effectively avoided.

[0098] The technical solution provided in this application monitors the network connection status with the server. When a network connection is detected to be broken, the current number of reconnections is determined. Based on this number of reconnections, the corresponding reconnection delay is determined, and a reconnection operation is initiated according to this delay. This technical solution, by dynamically calculating the reconnection delay, can quickly restore the connection when temporary network fluctuations occur (initially with fewer reconnections and shorter delays), ensuring the real-time performance of the service. However, when encountering continuous or severe network interruptions (with increasing reconnection counts and exponentially increasing delays), it can prevent clients from creating a "reconnection storm" on the server due to excessive reconnection attempts. This significantly reduces unnecessary load on the server during abnormal recovery periods, improves the stability and robustness of the entire system, and achieves an optimal balance between connection recovery efficiency and resource consumption in complex network environments.

[0099] See Figure 5This is a flowchart illustrating another embodiment of the data processing method provided in this application. Figure 5 The process shown is in Figure 1 , Figure 3 or Figure 4 Based on the illustrated process, a unified method for obtaining state data between the client and server is described. For example... Figure 5 As shown, the process may include the following steps: Step 501: Obtain the current status data. The status data may include, but is not limited to: data processing status, data interaction status with the server, and network connection status.

[0100] Step 502: Store the above-mentioned status data in a preset storage medium; the above-mentioned storage medium has a corresponding status interface, and the above-mentioned status interface provides the above-mentioned status data to the outside world in response to the received status query request.

[0101] The following provides a unified explanation of steps 501 and 502: The aforementioned data processing status refers to comprehensive information that reflects the current operational status of the data stream from reception and parsing to aggregation in real time. It typically includes dynamic indicators such as loading, parsing success / failure, data accumulation, and processing mode.

[0102] The aforementioned data interaction status with the server refers to the indicators of the real-time connectivity and activity of the SSE connection between the client and the server, specifically including phased states such as connection established, connecting, connection disconnected, and reconnection.

[0103] The aforementioned network connection status refers to the real-time connectivity status of the underlying network link between the client and the server.

[0104] In practical applications, the front-end interface displayed on the client often needs to obtain different states of the client or between the client and the server in order to update the components in the front-end interface. For example, when the preset data sent by the server is obtained, the data is displayed in the preset interface; when the preset data is not obtained, the data error is displayed on the front end.

[0105] Based on this, in this step, the current status data can be obtained in real time and stored in a preset storage medium. The storage medium may have a corresponding status interface, which can be used to provide the above status data to the outside world in response to the received status query request.

[0106] For example, a standardized StreamState can be predefined. <t>The interface can contain: data: T[]: a type-safe data array, loading: boolean: loading status, error: Error|null: error message, connected: boolean: connection status, and totalReceived: number: the number of received data. Based on the parameters contained in the above interface, the client can obtain the value of each parameter in real time and update the corresponding value to the corresponding position of the preset interface.

[0107] The technical solution provided in this application obtains current state data, which may include, but is not limited to, data processing status, data interaction status with the server, and network connection status. This state data is then stored in a preset storage medium. The storage medium has a corresponding state interface, which responds to received state query requests and provides the state data. This technical solution significantly reduces the development complexity and maintenance cost of real-time data stream applications by centrally encapsulating and standardizing the output of previously scattered and heterogeneous system operation status information (data processing, data interaction, and network connection). Specifically, this solution provides developers with a single, stable state query entry point, enabling UI components or business logic to conveniently obtain and respond to changes in all key states in a declarative and reactive manner. This eliminates the need to manually maintain and synchronize multiple independent state variables and event listeners throughout the code, thereby significantly reducing logical errors caused by inconsistent state management, improving code readability and modularity, and ultimately enhancing the development efficiency and system reliability of complex real-time applications.

[0108] See Figure 6 This is a block diagram illustrating an embodiment of a data processing apparatus provided in this application. As one embodiment, Figure 6 The device shown can be applied to the client side. For example... Figure 6 As shown, the device may include: The data segmentation module 61 is used to segment the data currently cached in the first preset cache area according to a preset boundary identifier to obtain multiple rows of data; the first preset cache area is used to continuously cache data fragments sent by the server. Data parsing module 62 is used to parse the row data for each row of data; The data removal module 63 is used to remove the row data from the first preset cache area when the row data is successfully parsed, so as to process the row data. The data retention module 64 is used to retain the row data in the first preset cache area in the event that the row data parsing fails, so as to merge it with the data subsequently received in the first preset cache area.

[0109] like Figure 7 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application, including a processor 71, a communication interface 72, a memory 73, and a communication bus 74. The processor 71, communication interface 72, and memory 73 communicate with each other via the communication bus 74. Memory 73 is used to store computer programs; In one embodiment of this application, when the processor 71 executes a program stored in the memory 73, it implements the data processing method provided in any of the foregoing method embodiments, including: For the data currently cached in the first preset cache area, the data is segmented according to a preset boundary identifier to obtain multiple rows of data; the first preset cache area is used to continuously cache data fragments sent by the server; For each row of data, the row of data is parsed; If the row data is successfully parsed, the row data is removed from the first preset cache area for data processing. If the parsing of the row data fails, the row data is retained in the first preset cache area for merging with subsequent data received in the first preset cache area.

[0110] This application also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the data processing method provided in any of the foregoing method embodiments.

[0111] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0112] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0113] It should be understood that the terminology used herein is for the purpose of specific illustrative implementations of the text only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "a," "an," and "described" as used herein may also mean including the plural forms. The terms "comprising," "including," "containing," and "having" are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in the specific order described or illustrated unless the order is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0114] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.< / t>

Claims

1. A data processing method, characterized in that, Applied to a client, the method includes: For the data currently cached in the first preset cache area, the data is segmented according to a preset boundary identifier to obtain multiple rows of data; the first preset cache area is used to continuously cache data fragments sent by the server; For each row of data, the row of data is parsed; If the row data is successfully parsed, the row data is removed from the first preset cache area for data processing. If the parsing of the row data fails, the row data is retained in the first preset cache area for merging with subsequent data received in the first preset cache area.

2. The method according to claim 1, characterized in that, Removing the row data from the first preset cache area to process the row data includes: Obtain the current data processing mode, which includes an incremental processing mode; When the data processing mode is incremental processing mode, the row data is transferred from the first preset cache area to the second preset cache area. When the successfully parsed row data meets the first preset condition, the row data stored in the second preset cache area is removed based on the storage time. The second preset cache area is used to store the row data to be processed.

3. The method according to claim 1, characterized in that, Removing the row data from the first preset cache area to process the row data includes: Obtain the current data processing mode, which includes batch processing mode; When the data processing mode is batch processing mode, the row data is transferred from the first preset cache area to the second preset cache area. When the successfully parsed row data meets the second preset condition, all row data in the second preset cache area is removed. The second preset cache area is used to store the row data to be processed.

4. The method according to claim 1, characterized in that, In the event that parsing the row data fails, the row data is retained in the first preset cache area, including: If parsing the row data fails, determine whether the row data is the last row of data; If it is determined that the row data is the last row data, the row data will be retained in the first preset cache area; If it is determined that the row data is not the last row, the row data is deleted from the first preset cache area.

5. The method according to claim 1, characterized in that, The server sends the data fragment via a network connection; the method further includes: Monitor the network connection status with the server; If the network connection is detected to be disconnected, determine the current number of reconnections; Based on the number of reconnections, determine the corresponding reconnection delay duration; Based on the reconnection delay duration, initiate a reconnection operation.

6. The method according to claim 5, characterized in that, Based on the number of reconnections, determine the current reconnection delay duration, including: The initial reconnection delay time is obtained by exponentiation of the number of reconnections to a preset value. The initial reconnection delay time is compared with the preset maximum reconnection delay time; The minimum value between the initial reconnection delay and the maximum reconnection delay is determined as the current reconnection delay.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Obtain current status data, which includes one or more of the following: data processing status, data interaction status with the server, and network connection status; The status data is stored in a preset storage medium; the storage medium has a corresponding status interface, and the status interface provides the status data to the outside world in response to a received status query request.

8. A data processing apparatus, characterized in that, Applied to a client, the device includes: The data segmentation module is used to segment the data currently cached in the first preset cache area according to a preset boundary identifier to obtain multiple rows of data; the first preset cache area is used to continuously cache data fragments sent by the server. The data parsing module is used to parse the data for each row. The data removal module is used to remove the row data from the first preset cache area when the row data is successfully parsed, so as to process the row data. The data retention module is used to retain the row data in the first preset cache area in the event that the row data parsing fails, so as to merge it with the data subsequently received in the first preset cache area.

9. An electronic device, characterized in that, include: A processor and a memory, the processor being configured to execute a data processing program stored in the memory to implement the data processing method according to any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data processing method according to any one of claims 1 to 7.

Images