Data processing method and device, equipment, readable storage medium and program product
By packaging and processing the target message based on the load status and fingerprint data, and generating the associated packaging unit, the problem of low processing efficiency of packaging unit in traditional technology is solved, and more efficient message parsing and storage optimization is achieved.
Patent Information
- Application Number
- CN202411284765.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-13
- Publication Date
- 2025-01-07
AI Technical Summary
The traditional message packaging and sending method results in low processing efficiency of the packaging unit and no correlation logic between messages, resulting in low efficiency of the target device during parsing.
The target message is packaged according to the current load state and fingerprint data, and a packaging unit containing the association relationship is generated. If the load state is high, the historical processing data will be used to estimate the processing time. If the load state is low, the packaging will be carried out according to the similarity to ensure the packaging unit Efficiency in target device resolution.
It improves the efficiency of the packaging unit when parsing the target device, reduces the repeated reading of the storage medium, reduces the difference in processing time, enhances the correlation between messages, and improves the overall processing efficiency.
Smart Images
Figure CN119276805A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a data processing method, apparatus, device, readable storage medium and program product. Background Art
[0002] Message packaging and sending technology is a technology based on load balancing between multiple storage nodes and servers. It can combine multiple messages into a packaged unit and transmit it to the target device, thereby improving the efficiency of message transmission.
[0003] The traditional message packaging and sending method usually packages multiple messages generated in the system into a packaging unit, and sends the packaging unit to the target device for processing.
[0004] However, the conventional method has the problem of low processing efficiency of the packaging unit. Summary of the invention
[0005] Based on this, it is necessary to provide a data processing method, device, equipment, readable storage medium and program product that can improve the processing efficiency of the packaging unit in response to the above technical problems.
[0006] In a first aspect, the present application provides a data processing method, comprising:
[0007] Acquire multiple target messages, where the target messages include fingerprint data;
[0008] According to the current load state and each fingerprint data, each target message is packaged to obtain a packaging unit;
[0009] Send the packaged unit to the target device.
[0010] In the above embodiment, when data is packaged for multiple target messages, the target messages are packaged based on the current load status and the fingerprint data corresponding to each target message. There is a certain correlation between the target messages contained in the resulting packaging unit, which makes the target device end more efficient in parsing the packaging unit, avoiding the problem of low packaging unit processing efficiency caused by directly packaging multiple messages in a preset order in traditional technologies, and there is no correlation logic between the messages in the packaging unit.
[0011] In one embodiment, according to the current load state and each fingerprint data, each target message is packaged to obtain a packaging unit, including:
[0012] If the load state is a high load state, the historical processing data corresponding to each fingerprint data is obtained, and the target messages are packaged according to each historical processing data to obtain a package unit, and the historical processing data is used to characterize the processing status of the historical messages corresponding to the fingerprint data;
[0013] If the load state is a low load state, each target message is packaged according to each fingerprint data to obtain a package unit.
[0014] In the above embodiments, corresponding message packaging processing is performed on each target message according to different load states, and the obtained packaging unit is more suitable for actual application scenarios.
[0015] In one embodiment, each target message is packaged according to each historical processing data to obtain a packaging unit, including:
[0016] For each fingerprint data, obtain the estimated processing time of the target message corresponding to the fingerprint data based on the historical processing data;
[0017] Message packaging is performed on multiple first messages in each target message whose estimated processing time meets the first condition to obtain a packaging unit, and the first condition at least includes that the time difference between the estimated processing times corresponding to each first message is less than the time difference threshold.
[0018] In the above embodiment, the processing time of each target message in the packaging unit obtained by the server is close, and the time required for the target device to parse the target message in the packaging unit is close, which improves the processing efficiency of the packaging unit.
[0019] In one embodiment, each target message is packaged according to each fingerprint data to obtain a packaging unit, including:
[0020] For each fingerprint data, obtain the similarity between the fingerprint data and other fingerprint data;
[0021] Message packaging is performed on multiple second messages whose similarities in each target message meet a second condition to obtain a packaging unit, wherein the second condition at least includes that a similarity difference between similarities corresponding to each second message is greater than a similarity difference threshold.
[0022] In the above embodiment, the target messages in the packaging unit sent by the server to the target device have a high correlation, and when parsing, there is no need to repeatedly read data corresponding to similar fingerprint data from the storage medium for parsing.
[0023] In one embodiment, the method further comprises:
[0024] Receiving the parsing result for the packaging unit sent by the target device;
[0025] Update the historical processing data according to the analysis results.
[0026] In the above embodiment, after obtaining the analysis result, the server can update the historical processing data of the fingerprint data corresponding to each target message according to the analysis result, so that the estimated processing time of the target message under high load state is more accurate.
[0027] In a second aspect, the present application provides a data processing method, comprising:
[0028] The package unit sent by the receiving server is obtained after the server performs message package processing on each target message according to the current load state and the fingerprint data corresponding to each target message.
[0029] In one embodiment, the method further comprises:
[0030] Parse the packaging unit to obtain the parsing result;
[0031] The analysis result is sent to the server, and the server is used to update the historical processing data corresponding to each fingerprint data according to the analysis result.
[0032] In a third aspect, the present application provides a data processing device, including:
[0033] An acquisition module, used for acquiring multiple target messages, wherein the target messages include fingerprint data;
[0034] A packing module is used to pack each target message according to the current load state and each fingerprint data to obtain a packing unit;
[0035] The sending module is used to send the packaging unit to the target device.
[0036] In a fourth aspect, the present application provides a data processing device, including:
[0037] The receiving module is used to receive the packaging unit sent by the server. The packaging unit is obtained after the server performs message packaging processing on each target message according to the current load state and the fingerprint data corresponding to each target message.
[0038] In a fifth aspect, an embodiment of the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the methods of the first and second aspects described above are implemented.
[0039] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method of the first aspect and the second aspect described above are implemented.
[0040] In a seventh aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the method in the first aspect and the second aspect are implemented.
[0041] The above-mentioned data processing method, device, equipment, readable storage medium and program product obtain multiple target messages, and then, according to the current load state and each fingerprint data, perform message packaging processing on each target message to obtain a packaging unit, and send the packaging unit to the target device, wherein the target message includes fingerprint data. In this way, when data packaging is performed on multiple target messages, the target messages are packaged based on the current load state and the fingerprint data corresponding to each target message. There is a certain correlation between the target messages contained in the obtained packaging unit, which makes the target device end more efficient in parsing the packaging unit, avoiding the problem of low efficiency of the packaging unit processing caused by directly packaging multiple messages in a preset order in the traditional technology, and there is no correlation logic between the messages in the packaging unit. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0043] Figure 1 This is an application environment diagram for message packaging in traditional technology;
[0044] Figure 2 This is a diagram of the internal structure of a packaging unit in traditional technology;
[0045] Figure 3 Another internal structure diagram of a packaging unit in the traditional technology;
[0046] Figure 4 An application environment diagram of a data processing method in an embodiment;
[0047] Figure 5 is a flow chart of a data processing method for a server in another embodiment;
[0048] Figure 6 A schematic diagram of a flow chart of a message packaging process in another embodiment;
[0049] Figure 7 is a flowchart of an exemplary data processing method in one embodiment;
[0050] Figure 8is a structural block diagram of a data processing device for a server in one embodiment;
[0051] Fig. 9 is a structural block diagram of a data processing device for a target device in one embodiment;
[0052] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0054] Message packaging and sending technology is a technology based on load balancing between multiple storage nodes and servers. It usually packages messages and distributes them to different servers for processing based on load balancing strategies (such as polling, least connections, hashing, etc.), thereby evenly distributing the data processing tasks corresponding to the messages to each server, avoiding overloading of a particular server.
[0055] The traditional message packaging and sending method usually packages multiple messages into a packaging unit and sends the packaging unit to the target device for processing. Figure 1 ,After receiving multiple messages through other devices, such as a terminal, server 1 integrates the multiple messages into a packaged unit and sends it to server 2 for parsing and processing.
[0056] However, the traditional message packaging and sending method has the problem of low processing efficiency of the packaging unit. For details, please refer to the following content:
[0057] 1. The parsing time of a packaged unit depends on the longest message. Figure 2 There are four messages in the packaging unit 1. After the packaging unit is sent to the other end, the processing time of messages Msg1 to Msg3 is in the single digit, and the processing can be completed quickly. However, the processing time of Msg4 is 500ms, so the packaging unit 1 can only be released after 500ms, and the processing efficiency is low.
[0058] 2. There is no correlation between the messages in the packaging unit. When messages with repeated data characteristics are in multiple packaging units, the target device will continuously solidify the messages with repeated data characteristics when processing the messages in the packaging unit. For some target devices with the data deduplication feature enabled, data with the same data characteristics will be continuously read from the disk, increasing the number of disk reads and writes and reducing the disk life.
[0059] Data deduplication is a core data management technology that is mainly used to reduce storage consumption and improve data processing efficiency. It optimizes the use of storage space by identifying and removing duplicate data fragments and replacing duplicate data with references to existing data.
[0060] Regarding the target device's data deduplication process for messages in the packaging unit, it usually identifies the fingerprint data corresponding to the message and determines whether there is data with the same fingerprint data in the storage medium (such as a disk) based on the fingerprint data. If found, the corresponding data is read from the storage medium and compared with the data in the current message. The deletion or retention of the data is determined based on the comparison result, thereby avoiding duplicate data in the storage inoculation and reducing the waste of storage resources.
[0061] Reference Figure 3 , four messages Data1 to Data4 are packaged in packaging unit 1, and the corresponding data characteristics (fingerprint data) are FP1-FP4 respectively. When the target device receives packaging unit 1, it will process each message accordingly according to the fingerprint data of each message, and solidify the four messages in the storage medium. Subsequently, the target device receives packaging unit 2, and four messages Data5 to Data8 are packaged in packaging unit 2. Among them, the fingerprint data of message Data5 is the same as that of Data1, and the fingerprint data of message Data6 is the same as that of Data2. The target device needs to repeatedly read the data corresponding to FP1 and the data corresponding to FP2 and compare them with the data of messages Data5 and Data6.
[0062] In view of this, the present application provides a data processing method, device, equipment, readable storage medium and program product, which obtain multiple target messages, and then, according to the current load state and each fingerprint data, perform message packaging processing on each target message to obtain a packaging unit, and send the packaging unit to the target device, wherein the target message includes the fingerprint data. In this way, when data is packaged for multiple target messages, the target messages are packaged based on the current load state and the fingerprint data corresponding to each target message, and there is a certain correlation between the target messages contained in the obtained packaging unit, so that the target device end is more efficient in parsing the packaging unit, avoiding the problem of low efficiency of the packaging unit processing caused by directly packaging multiple messages in a preset order in the traditional technology, and there is no correlation logic between the messages in the packaging unit.
[0063] The data processing method provided in the embodiment of the present application can be applied to Figure 4In the application environment shown. Among them, multiple devices 102 communicate with the server 101 through the network, and the target device can be any device 102. The data storage system can store the data that the server 101 needs to process. The data storage system can be integrated on the server 101, or it can be placed on the cloud or other network servers. Among them, the device 102 is a device with data processing capabilities, which can be a server or a terminal. The server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.
[0064] In an exemplary embodiment, Figure 5 As shown, a data processing method is provided, which is applied to Figure 4 The server in the example is used to illustrate, including the following steps 501 to 503. Among them:
[0065] Step 501, obtaining multiple target messages.
[0066] The message may be a data processing task to be processed, and the target message may be a message to be packaged and sent to a target device.
[0067] Optionally, the server may obtain input messages through other external devices; optionally, the server may obtain messages generated by the system itself.
[0068] In an embodiment of the present application, after the server obtains the message, it will calculate the fingerprint data of each message according to a preset calculation method. After obtaining the fingerprint data of each message, the server can determine the target message to be packaged and sent to the target device based on the fingerprint data, wherein the target message includes the fingerprint data.
[0069] Step 502: Packaging each target message according to the current load state and each fingerprint data to obtain a packaging unit.
[0070] After determining multiple target messages, the server can obtain the current load status, which is used to characterize the idleness of the computing resources of the current server. In an embodiment of the present application, the load status may include a high load status and a low load status, wherein the high load status is used to characterize that the computing resources of the current server are very tight, and the low load status can characterize that the computing resources of the current server are idle.
[0071] The server can determine the current load status through load usage data such as GPU occupancy, CPU occupancy, and bandwidth occupancy.
[0072] Under different load states, the emphasis is different when processing the packaging unit. For example, when the current load state is a high load state, the packaging unit needs to be processed and parsed quickly. When the current covering wall is in a low load state, it can be considered to minimize the reading of the storage medium when processing the packaging unit. Therefore, in an embodiment of the present application, the server can package each target message according to the load state and fingerprint data to meet different actual application needs.
[0073] In one possible implementation, if the load state is a high load state, the server can obtain the processing time of the historical messages corresponding to each fingerprint data based on each fingerprint data, and then estimate the processing time of each target message based on the processing time of each historical message to obtain the estimated processing time. The server can package the target messages with similar estimated processing times; in another possible implementation, if the load state is a low load state, the server can use the fingerprint data as the data feature of each target message, and calculate the similarity between each fingerprint data. Then, the server can package the target messages corresponding to the fingerprint data with high similarity.
[0074] Step 503: Send the packaged unit to the target device.
[0075] After obtaining the packaging unit, the server may first determine the target device, and then send the packaging unit to the target device based on a preset network communication protocol, so that the target device can parse and process the packaging unit.
[0076] In a possible implementation, after the target device parses the packaging unit, the server may also receive a parsing result fed back by the target device, thereby determining whether the packaging unit is successfully sent to the target device according to the parsing result.
[0077] In the above embodiment, when data is packaged for multiple target messages, the target messages are packaged based on the current load status and the fingerprint data corresponding to each target message. There is a certain correlation between the target messages contained in the resulting packaging unit, which makes the target device end more efficient in parsing the packaging unit, avoiding the problem of low packaging unit processing efficiency caused by directly packaging multiple messages in a preset order in traditional technologies, and there is no correlation logic between the messages in the packaging unit.
[0078] In one embodiment, based on the above Figure 5 The embodiment shown in the figure is related to the process of performing message packaging processing on each target message according to the current load state and each fingerprint data to obtain a packaging unit. Figure 6 As shown, in this embodiment, step 502 includes Figure 6 Step 601 and step 602 are shown.
[0079] Step 601: If the load state is a high load state, the historical processing data corresponding to each fingerprint data is obtained, and message packaging processing is performed on each target message according to each historical processing data to obtain a packaging unit.
[0080] The historical processing data is used to characterize the processing status of the historical messages corresponding to the fingerprint data.
[0081] In an embodiment of the present application, historical processing data may include the processing time of historical messages corresponding to the fingerprint data, the corresponding data type (such as metadata or ordinary data), the degree of hotness and coldness, the frequency of data access, etc.
[0082] In one possible implementation, for each fingerprint data, the server can obtain the estimated processing time of the target message corresponding to the fingerprint data based on the historical processing data, and perform message packaging processing on multiple first messages in each target message whose estimated processing time meets the first condition to obtain a packaging unit.
[0083] Among them, the first condition at least includes that the duration difference between the estimated processing times corresponding to the first messages is less than the duration difference threshold.
[0084] In an embodiment of the present application, the service can estimate the processing time of the target message corresponding to the fingerprint data based on the historical processing data. Optionally, the server can perform weighted average calculation on the processing time, the corresponding data type (such as metadata or ordinary data), the degree of hotness or coldness, the data access frequency and other data according to preset weights to obtain the estimated processing time corresponding to the target message; optionally, the server can input the historical processing data into a pre-trained estimation model to obtain the estimated processing time output by the estimation model.
[0085] After obtaining the estimated processing time corresponding to each target message, the server can perform message packaging processing on multiple first messages whose estimated processing time satisfies the first condition in each target message.
[0086] In an embodiment of the present application, the server can calculate the difference between each estimated processing time, and the first condition can be that the difference between the estimated processing times is less than a preset difference threshold. The difference between the estimated processing times is less than the preset difference threshold, which can indicate that the processing times required for the corresponding two target messages are close.
[0087] In this way, the processing time of each target message in the obtained packaging unit is close, and the time required for the target device to parse the target message in the packaging unit is close, thereby improving the processing efficiency of the packaging unit.
[0088] Step 602: If the load state is a low load state, each target message is packaged according to each fingerprint data to obtain a package unit.
[0089] In an embodiment of the present application, if the load state is a low load state, the focus of the actual application requirements may be on reducing the number of times the storage medium is read, then the server may find the correlation between the target messages and package the target messages with higher correlation.
[0090] In a possible implementation, if the load state is a low load state, the server can obtain the similarity between each fingerprint data and other fingerprint data, and then perform message packaging processing on multiple second messages in each target message whose similarity meets the second condition to obtain a packaging unit.
[0091] In an embodiment of the present application, the server may use the fingerprint data as the data feature of the target message, and then obtain the similarity between the fingerprint data and other fingerprint data. The higher the similarity, the higher the degree of association between the corresponding target messages.
[0092] Optionally, the server may calculate the similarity between the fingerprint data and other fingerprint data according to preset conditions; optionally, the server may input the fingerprint data into a pre-trained similarity calculation model, thereby obtaining the similarity between each fingerprint data output by the model.
[0093] After obtaining the similarities between the fingerprint data, the server can package the target messages according to the second condition and the similarities. In the embodiment of the present application, the second condition at least includes that the similarity difference between the similarities corresponding to the second messages is greater than the similarity difference threshold.
[0094] In this way, the target messages in the packaging unit sent by the server to the target device have a high correlation, and when parsing, there is no need to repeatedly read data corresponding to similar fingerprint data from the storage medium for parsing.
[0095] In one possible implementation, if the load state is a high load state, the server can first obtain the similarity between each fingerprint data, and determine the similar fingerprint data whose similarity meets the second condition. Then, the server can determine the estimated processing time of the target message corresponding to each similar fingerprint data based on the historical processing data corresponding to each similar fingerprint data, and determine the target message corresponding to the similar fingerprint data whose estimated processing time meets the first condition. The server can package these target messages for message processing. In this way, the estimated processing time of the target messages in the obtained packaged unit is similar and highly correlated, so that the target device has higher processing efficiency for the packaged unit.
[0096] In an embodiment of the present application, after the server sends the packaging unit to the target device, it can also receive the analysis result for the packaging unit sent by the target device, and then update the data of each historical processing data according to the analysis result.
[0097] In an embodiment of the present application, the parsing result may include processing data corresponding to each target message, and the processing data is related to the historical processing data, including at least the processing time of the target message, the corresponding data type (such as metadata or ordinary data), the degree of hotness and coldness, the frequency of data access, etc. After obtaining the parsing result, the server can update the historical processing data of the fingerprint data corresponding to each target message according to the parsing result, so that the estimated processing time of the target message under subsequent high load conditions is more accurate.
[0098] In the above embodiments, corresponding message packaging processing is performed on each target message according to different load states, and the obtained packaging unit is more suitable for actual application scenarios.
[0099] In one embodiment, a data processing method is provided. Figure 4 Taking any device 402 (i.e., the target device) in the example as an example, the following steps are included:
[0100] Receives the packet unit sent by the server.
[0101] The packing unit is obtained after the server packs each target message according to the current load status and the fingerprint data corresponding to each target message.
[0102] In an embodiment of the present application, the target device can receive a packaging unit sent by the server. The process of obtaining the packaging unit can refer to the above embodiment and will not be repeated here.
[0103] In a possible implementation, after acquiring the packaging unit sent by the server, the target device may parse the packaging unit to obtain a parsing result, and send the parsing result to the server.
[0104] The analysis result is used by the server to update the historical processing data corresponding to each fingerprint data according to the analysis result.
[0105] In an embodiment of the present application, when the target device parses the packaging unit, it can generate the processing time corresponding to each target message in the packaging unit, the corresponding data type (such as metadata or ordinary data), the degree of hotness and coldness, the data access frequency, etc. The target device can carry these data in the parsing result and send the parsing result to the server, so that the server can update the historical processing data corresponding to each fingerprint data according to the parsing result.
[0106] In one embodiment, please refer to Figure 7 , which shows a flowchart of an exemplary data processing method provided in an embodiment of the present application, which can be applied to Figure 4 In the implementation environment shown.
[0107] Step 701: The server obtains multiple target messages.
[0108] The target message includes fingerprint data.
[0109] Step 702: If the load state is a high load state, the server obtains the historical processing data corresponding to each fingerprint data.
[0110] Step 703: for each fingerprint data, the server obtains the estimated processing time of the target message corresponding to the fingerprint data according to the historical processing data.
[0111] Step 704: The server performs message packaging processing on a plurality of first messages whose estimated processing time meets the first condition in each target message to obtain a packaging unit.
[0112] Among them, the first condition at least includes that the duration difference between the estimated processing times corresponding to the first messages is less than the duration difference threshold.
[0113] The historical processing data is used to characterize the processing status of the historical messages corresponding to the fingerprint data.
[0114] Step 705: If the load state is a low load state, the server obtains the similarity between each fingerprint data and other fingerprint data.
[0115] Step 706: The server performs message packaging processing on multiple second messages whose similarities satisfy the second condition in each target message to obtain a packaging unit.
[0116] The second condition at least includes that a similarity difference between similarities corresponding to each second message is greater than a similarity difference threshold.
[0117] Step 707: The server sends the packaged unit to the target device.
[0118] The packing unit is obtained after the server packs each target message according to the current load status and the fingerprint data corresponding to each target message.
[0119] Step 708: The target device parses the packaging unit to obtain a parsing result.
[0120] Step 709: The target device sends the parsing result to the server.
[0121] The analysis result is used by the server to update the historical processing data corresponding to each fingerprint data according to the analysis result.
[0122] Step 710: The server receives the parsing result for the packaging unit sent by the target device.
[0123] Step 711: The server updates each historical processing data according to the analysis result.
[0124] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0125] Based on the same inventive concept, the embodiment of the present application also provides a data processing device for implementing the data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more data processing device embodiments provided below can refer to the limitations on the data processing method above, and will not be repeated here.
[0126] In an exemplary embodiment, Figure 8 As shown, a data processing device is provided, including: an acquisition module 801, a packaging module 802 and a sending module 803, wherein:
[0127] An acquisition module 801 is used to acquire multiple target messages, wherein the target messages include fingerprint data;
[0128] The packing module 802 is used to pack each target message according to the current load state and each fingerprint data to obtain a packing unit;
[0129] The sending module 803 is used to send the packaging unit to a target device.
[0130] In one embodiment, the packaging module 802 includes:
[0131] A high-load packaging unit, used for obtaining historical processing data corresponding to each fingerprint data if the load state is a high-load state, and performing message packaging processing on each target message according to each historical processing data to obtain the packaging unit, wherein the historical processing data is used to characterize the processing status of the historical messages corresponding to the fingerprint data;
[0132] A low-load packaging unit is used to, if the load state is a low-load state, perform message packaging processing on each of the target messages according to each of the fingerprint data to obtain the packaging unit.
[0133] In one embodiment, the high load packaging unit is specifically configured to perform:
[0134] For each of the fingerprint data, obtaining an estimated processing time of a target message corresponding to the fingerprint data according to the historical processing data;
[0135] Message packaging is performed on multiple first messages in each of the target messages whose estimated processing time satisfies a first condition to obtain the packaging unit, wherein the first condition at least includes that the time difference between the estimated processing time corresponding to each of the first messages is less than a time difference threshold.
[0136] In one embodiment, the low load packaging unit is specifically configured to perform:
[0137] For each fingerprint data, obtaining the similarity between the fingerprint data and other fingerprint data;
[0138] Message packaging is performed on multiple second messages whose similarities in each of the target messages meet a second condition to obtain the packaging unit, wherein the second condition at least includes that a similarity difference between the similarities corresponding to each of the second messages is greater than a similarity difference threshold.
[0139] In one embodiment, the data processing device further comprises:
[0140] A result receiving module, used for receiving the parsing result for the packaging unit sent by the target device;
[0141] A data updating module is used to update each of the historical processing data according to the analysis result.
[0142] In an exemplary embodiment, Fig. 9 As shown, a data processing device is provided, including: a receiving module 901, wherein:
[0143] The receiving module 901 is used to receive a packaging unit sent by a server, where the packaging unit is obtained by the server after packaging each target message according to a current load state and fingerprint data corresponding to each target message.
[0144] In one embodiment, the data processing device further comprises:
[0145] A parsing module, used to parse the packaging unit to obtain a parsing result;
[0146] The result sending module is used to send the analysis result to the server, and the analysis result is used by the server to update the historical processing data corresponding to each fingerprint data according to the analysis result.
[0147] Each module in the above data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.
[0148] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a device. The internal structure diagram thereof may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data processing data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.
[0149] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0150] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:
[0151] Acquire multiple target messages, wherein the target messages include fingerprint data;
[0152] According to the current load state and each of the fingerprint data, each of the target messages is packaged to obtain a packaging unit;
[0153] The packaged unit is sent to a target device.
[0154] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0155] If the load state is a high load state, the historical processing data corresponding to each of the fingerprint data is obtained, and the target messages are packaged according to the historical processing data to obtain the packaging unit, wherein the historical processing data is used to characterize the processing status of the historical messages corresponding to the fingerprint data;
[0156] If the load state is a low load state, each of the target messages is packaged according to each of the fingerprint data to obtain the packaging unit.
[0157] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0158] For each of the fingerprint data, obtaining an estimated processing time of a target message corresponding to the fingerprint data according to the historical processing data;
[0159] Message packaging is performed on multiple first messages in each of the target messages whose estimated processing time satisfies a first condition to obtain the packaging unit, wherein the first condition at least includes that the time difference between the estimated processing time corresponding to each of the first messages is less than a time difference threshold.
[0160] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0161] For each fingerprint data, obtaining the similarity between the fingerprint data and other fingerprint data;
[0162] Message packaging is performed on multiple second messages whose similarities in each of the target messages meet a second condition to obtain the packaging unit, wherein the second condition at least includes that a similarity difference between the similarities corresponding to each of the second messages is greater than a similarity difference threshold.
[0163] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0164] Receiving a parsing result for the packaging unit sent by the target device;
[0165] The historical processing data are updated according to the analysis result.
[0166] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:
[0167] A packaging unit sent by a receiving server is obtained after the server performs message packaging processing on each target message according to a current load state and fingerprint data corresponding to each target message.
[0168] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0169] Parsing the packaging unit to obtain a parsing result;
[0170] The analysis result is sent to the server, and the server is used to update the historical processing data corresponding to each fingerprint data according to the analysis result.
[0171] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
[0172] Acquire multiple target messages, wherein the target messages include fingerprint data;
[0173] According to the current load state and each of the fingerprint data, each of the target messages is packaged to obtain a packaging unit;
[0174] The packaged unit is sent to a target device.
[0175] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0176] If the load state is a high load state, the historical processing data corresponding to each of the fingerprint data is obtained, and the target messages are packaged according to the historical processing data to obtain the packaging unit, wherein the historical processing data is used to characterize the processing status of the historical messages corresponding to the fingerprint data;
[0177] If the load state is a low load state, each of the target messages is packaged according to each of the fingerprint data to obtain the packaging unit.
[0178] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0179] For each of the fingerprint data, obtaining an estimated processing time of a target message corresponding to the fingerprint data according to the historical processing data;
[0180] Message packaging is performed on multiple first messages in each of the target messages whose estimated processing time satisfies a first condition to obtain the packaging unit, wherein the first condition at least includes that the time difference between the estimated processing time corresponding to each of the first messages is less than a time difference threshold.
[0181] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0182] For each fingerprint data, obtaining the similarity between the fingerprint data and other fingerprint data;
[0183] Message packaging is performed on multiple second messages whose similarities in each of the target messages meet a second condition to obtain the packaging unit, wherein the second condition at least includes that a similarity difference between the similarities corresponding to each of the second messages is greater than a similarity difference threshold.
[0184] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0185] Receiving a parsing result for the packaging unit sent by the target device;
[0186] The historical processing data are updated according to the analysis result.
[0187] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
[0188] A packaging unit sent by a receiving server is obtained after the server performs message packaging processing on each target message according to a current load state and fingerprint data corresponding to each target message.
[0189] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0190] Parsing the packaging unit to obtain a parsing result;
[0191] The analysis result is sent to the server, and the server is used to update the historical processing data corresponding to each fingerprint data according to the analysis result.
[0192] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:
[0193] Acquire multiple target messages, wherein the target messages include fingerprint data;
[0194] According to the current load state and each of the fingerprint data, each of the target messages is packaged to obtain a packaging unit;
[0195] The packaged unit is sent to a target device.
[0196] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0197] If the load state is a high load state, the historical processing data corresponding to each of the fingerprint data is obtained, and the target messages are packaged according to the historical processing data to obtain the packaging unit, wherein the historical processing data is used to characterize the processing status of the historical messages corresponding to the fingerprint data;
[0198] If the load state is a low load state, each of the target messages is packaged according to each of the fingerprint data to obtain the packaging unit.
[0199] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0200] For each of the fingerprint data, obtaining an estimated processing time of a target message corresponding to the fingerprint data according to the historical processing data;
[0201] Message packaging is performed on multiple first messages in each of the target messages whose estimated processing time satisfies a first condition to obtain the packaging unit, wherein the first condition at least includes that the time difference between the estimated processing time corresponding to each of the first messages is less than a time difference threshold.
[0202] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0203] For each fingerprint data, obtaining the similarity between the fingerprint data and other fingerprint data;
[0204] Message packaging is performed on multiple second messages whose similarities in each of the target messages meet a second condition to obtain the packaging unit, wherein the second condition at least includes that a similarity difference between the similarities corresponding to each of the second messages is greater than a similarity difference threshold.
[0205] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0206] Receiving a parsing result for the packaging unit sent by the target device;
[0207] The historical processing data are updated according to the analysis result.
[0208] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:
[0209] A packaging unit sent by a receiving server is obtained after the server performs message packaging processing on each target message according to a current load state and fingerprint data corresponding to each target message.
[0210] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0211] Parsing the packaging unit to obtain a parsing result;
[0212] The analysis result is sent to the server, and the server is used to update the historical processing data corresponding to each fingerprint data according to the analysis result.
[0213] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.
[0214] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.
[0215] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0216] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A data processing method, characterized in that: The method comprises: Acquire multiple target messages, wherein the target messages include fingerprint data; According to the current load state and each of the fingerprint data, each of the target messages is packaged to obtain a packaging unit; The packaged unit is sent to a target device.
2. The method according to claim 1, characterized in that The process of performing message packaging processing on each of the target messages according to the current load state and each of the fingerprint data to obtain a packaging unit includes: If the load state is a high load state, the historical processing data corresponding to each of the fingerprint data is obtained, and the target messages are packaged according to the historical processing data to obtain the packaging unit, wherein the historical processing data is used to characterize the processing status of the historical messages corresponding to the fingerprint data; If the load state is a low load state, each of the target messages is packaged according to each of the fingerprint data to obtain the packaging unit.
3. The method according to claim 2, characterized in that The process of performing message packaging processing on each of the target messages according to each of the historical processing data to obtain the packaging unit includes: For each of the fingerprint data, obtaining an estimated processing time of a target message corresponding to the fingerprint data according to the historical processing data; Message packaging is performed on multiple first messages in each of the target messages whose estimated processing time satisfies a first condition to obtain the packaging unit, wherein the first condition at least includes that the time difference between the estimated processing time corresponding to each of the first messages is less than a time difference threshold.
4. The method according to claim 2, characterized in that: The step of performing message packaging processing on each of the target messages according to each of the fingerprint data to obtain the packaging unit includes: For each fingerprint data, obtaining the similarity between the fingerprint data and other fingerprint data; Message packaging is performed on multiple second messages whose similarities in each of the target messages meet a second condition to obtain the packaging unit, wherein the second condition at least includes that a similarity difference between the similarities corresponding to each of the second messages is greater than a similarity difference threshold.
5. The method according to claim 2, characterized in that: The method further comprises: Receiving a parsing result for the packaging unit sent by the target device; The historical processing data are updated according to the analysis result.
6. A data processing method, characterized in that: The method comprises: A packaging unit sent by a receiving server is obtained after the server performs message packaging processing on each target message according to a current load state and fingerprint data corresponding to each target message.
7. The method according to claim 6, characterized in that The method further comprises: Parsing the packaging unit to obtain a parsing result; The analysis result is sent to the server, and the server is used to update the historical processing data corresponding to each fingerprint data according to the analysis result.
8. A data processing device, characterized in that: The device comprises: An acquisition module, used for acquiring a plurality of target messages, wherein the target messages include fingerprint data; A packing module, used for packing each of the target messages according to the current load state and each of the fingerprint data to obtain a packing unit; The sending module is used to send the packaging unit to the target device.
9. A data processing device, characterized in that: The device comprises: The receiving module is used to receive the packaging unit sent by the server, and the packaging unit is obtained after the server performs message packaging processing on each target message according to the current load state and the fingerprint data corresponding to each target message.
10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.