Data processing method and device, equipment and storage medium

Abstract

The present disclosure provides a data processing method and device, equipment and storage medium, relates to the technical field of computers, in particular to the technical field of information retrieval. The specific implementation scheme is: receiving a remote service request from an upstream operator node; analyzing the remote service request to obtain to-be-processed data; sending the to-be-processed data to a computing server; receiving a computing result of the to-be-processed data from the computing server; generating target data according to the computing result and the to-be-processed data; and sending the target data to a downstream operator node corresponding to the upstream operator node.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computers, and particularly relates to the technical field of information retrieval. BACKGROUND

[0002] Stream computing technology refers to real-time processing of a continuously generated data stream. Compared with batch processing, stream computing has higher timeliness. A stream computing system is a computing system that can process stream computing. The stream computing system can include various operators.

[0003] Currently, search engines and recommendation engines need to process and process original data when guiding high-quality content data, such as calculating summaries of news, recognizing image texts of pictures, calculating features of commodity pictures or commodity videos, and the like, so as to obtain results more in line with user search intentions, so as to improve overall search experience. In the process of establishing a database for original data, various operators in the stream computing system can be used to process and process the original data in batches, so as to meet different business scenario requirements. SUMMARY

[0004] The present disclosure provides a data processing method, device, equipment, storage medium and program product.

[0005] According to an aspect of the present disclosure, a data processing method is provided, including: receiving a remote service request from an upstream operator node; parsing the remote service request to obtain to-be-processed data; sending the to-be-processed data to a computing server; receiving a computing result of the computing server for the to-be-processed data; generating target data according to the computing result and the to-be-processed data; and sending the target data to a downstream operator node corresponding to the upstream operator node.

[0006] According to another aspect of the present disclosure, a data processing apparatus is provided, including: a first receiving module configured to receive a remote service request from an upstream operator node; a parsing module configured to parse the remote service request to obtain to-be-processed data; a first sending module configured to send the to-be-processed data to a computing server; a second receiving module configured to receive a computing result of the computing server for the to-be-processed data; a generating module configured to generate target data according to the computing result and the to-be-processed data; and a second sending module configured to send the target data to a downstream operator node corresponding to the upstream operator node.

[0007] Another aspect of the present disclosure provides an electronic device, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method shown in the embodiments of the present disclosure.

[0008] According to another aspect of the embodiments of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to make the computer perform the method shown in the embodiments of the present disclosure.

[0009] According to another aspect of the embodiments of the present disclosure, a computer program product is provided, comprising computer programs / instructions, characterized by, when the computer programs / instructions are executed by a processor, implementing the steps of the method shown in the embodiments of the present disclosure.

[0010] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings are used to better understand the present scheme, and do not limit the present disclosure. Among them:

[0012] Figure 1 An exemplary system architecture to which the data processing method and device according to the embodiments of the present disclosure can be applied is schematically shown;

[0013] Figure 2 A flowchart of a data processing method according to an embodiment of the present disclosure is schematically shown;

[0014] Figure 3 A schematic diagram of sending a target request to a computing server according to an embodiment of the present disclosure is schematically shown;

[0015] Figure 4 A flowchart of a data processing method according to another embodiment of the present disclosure is schematically shown;

[0016] Figure 5 A schematic diagram of a data processing method according to another embodiment of the present disclosure is schematically shown;

[0017] Figure 6 A block diagram of a data processing device according to an embodiment of the present disclosure is schematically shown; and

[0018] Figure 7 A block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0019] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are presented for the purpose of illustration and description. It is to be understood that the embodiments described herein are exemplary only, and various changes and modifications can be made thereto without departing from the scope and spirit of the present disclosure. As such, the exemplary embodiments described herein are not intended to be exhaustive or to be unduly restrictive of the scope of the present disclosure. Accordingly, those skilled in the art will appreciate that the embodiments described herein can be practiced with various modifications and alterations, and that the disclosure is not limited to the embodiments described herein, but rather, the intent is that the disclosure be given full scope as defined in the claims.

[0020] The following will be described in conjunction with Figure 1 The application scenarios of the data processing method and device provided by the present disclosure are described.

[0021] Figure 1 An exemplary system architecture 100 to which the data processing method and device according to embodiments of the present disclosure can be applied is schematically shown. It should be noted that, Figure 1 The system architecture shown is only an example of the system architecture to which embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but does not mean that embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0022] As Figure 1 The system architecture 100 according to the embodiment can include a plurality of operator nodes and a computing server, such as operator nodes 101, 102, 103, 104, an asynchronous operator 105 and a computing server 106, as shown.

[0023] The operator nodes 101, 102, 103, 104 can perform stream computing tasks, such as picture feature calculation, text recognition, link replacement, picture cropping, watermark addition, etc.

[0024] The computing server 106 can be used to provide remote services, which can include, for example, news summarization calculation, picture text recognition, feature calculation on commodity pictures or commodity videos, etc.

[0025] According to embodiments of the present disclosure, when the operator nodes 101, 102, 103, 104 perform stream computing tasks, external services can be used to assist in performing the stream computing tasks. For example, the node 103 can access the computing server 106 to obtain remote services provided by the computing server 106.

[0026] For example, the operator node 102 can send the data to be processed to the computing server 106, so that the computing server 106 performs calculation on the data to be processed to obtain a calculation result. The operator node 103 can continue subsequent calculation according to the calculation result.

[0027] According to an embodiment of the present disclosure, an asynchronous operator 105 can be arranged between the operator node 102 and the operator node 103. The asynchronous operator 105 processes the remote service request of the operator node to the computing server 106. Based on this, the operator node 102 is upstream of the asynchronous operator 105, and is an upstream operator node of the asynchronous operator 105, and the operator node 103 is downstream of the asynchronous operator 105, and is a downstream operator node of the asynchronous operator 105.

[0028] The remote computing resource provided by the computing server 106 is limited, and in the case where the operator node 102 is a multi-instance deployment, the computing server 106 needs to process more requests, and the instantaneous traffic is high, which can cause the remote service to be unstable. Based on this, the asynchronous operator 105 can receive the remote service request from each upstream operator node 102. The remote service request is parsed to obtain to-be-processed data. Then the to-be-processed data is sent to the computing server 106. Then the computing result of the computing server 106 for the to-be-processed data is received. According to the computing result and the to-be-processed data, target data is generated. Then the target data is sent to the downstream operator node 103 corresponding to the operator node 102.

[0029] By transferring the remote service request of each operator node 102 to the asynchronous operator node 105 for processing, the upstream operator nodes 102 can be uniformly controlled, and the threat to the stability of the remote service caused by the high instantaneous traffic of the business operator can be reduced.

[0030] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information comply with relevant laws and regulations, necessary security measures are taken, and do not violate public order and good customs.

[0031] In the technical solution of the present disclosure, the authorization or consent of the user is obtained before the user's personal information is acquired or collected.

[0032] The following will be described in conjunction with Figure 2 The data processing method provided by the present disclosure is described.

[0033] Figure 2 A flowchart of a data processing method according to an embodiment of the present disclosure is schematically shown.

[0034] As Figure 2 shown, the data processing method 200 includes operations S210-S260. The method 200 can be performed by the asynchronous operator shown above, for example.

[0035] In operation S210, a remote service request from an upstream operator node is received.

[0036] According to embodiments of this disclosure, the upstream operator node can be, for example, an operator node that intends to request remote services. This operator node can be set as the upstream operator node of an asynchronous operator, so that the remote service request of this operator node is sent to the asynchronous node for unified processing.

[0037] According to embodiments of this disclosure, a remote service request can be used to request remote services, which may include, for example, services such as summarizing news, recognizing text in images, and performing feature calculations on product images or videos.

[0038] Then, by operating S220, the remote service request is parsed to obtain the data to be processed.

[0039] According to embodiments of this disclosure, at least one preset field path can be pre-set for each type of remote service request. Based on this, the corresponding field in the remote service request that matches each preset field path can be determined as data to be processed.

[0040] In operation S230, the data to be processed is sent to the computing server.

[0041] According to embodiments of this disclosure, for example, the computing server corresponding to the remote service request can be determined, and then a target request conforming to the communication specifications of the computing server can be determined based on the data to be processed. The target request is then sent to the computing server to request the computing server to perform corresponding data processing on the data to be processed.

[0042] For example, in this embodiment, a common request field corresponding to the remote service request can be obtained. This common request field can be determined according to the communication specifications of the computing server. Then, a target request can be generated based on the original data to be processed and the common request field. The target request is then sent to the computing server.

[0043] In operation S240, the calculation results for the data to be processed are received from the computing server.

[0044] In operation S250, target data is generated based on the calculation results and the data to be processed.

[0045] According to embodiments of this disclosure, for example, the calculation results can be added to the data to be processed to obtain the target data.

[0046] In operation S260, the target data is sent to the downstream operator node corresponding to the upstream operator node.

[0047] According to an embodiment of the present disclosure, a downstream operator node can be an operator node downstream of and connected to an upstream operator node. The downstream operator node can perform subsequent computing operations according to a computing result of the data to be processed.

[0048] According to an embodiment of the present disclosure, by having an asynchronous operator node process remote service requests of various operator nodes, unified traffic control can be performed on the upstream operator nodes, and the threat of high instantaneous traffic of a business operator to the stability of a remote service can be reduced.

[0049] According to another embodiment of the present disclosure, a configuration file can be preset for each type of remote service request. The configuration file can include, for example, at least one field path, a write-back operator node identifier, a writing mode, a common request field, and the like. The configuration file can include, for example, a schema.

[0050] The at least one field path can be used to indicate which fields in the remote service request need to be requested to be processed by a computing server. Based on this, the remote service request can be parsed according to the configuration file to determine corresponding fields in the remote service request that match each preset field path as data to be processed.

[0051] The write-back operator node identifier can be used to indicate to which operator node the computing result should be returned. Based on this, a downstream operator node corresponding to the upstream operator node can be determined according to the write-back operator node identifier in the configuration file.

[0052] The writing mode can be used to indicate a way in which the computing result should be added to the data to be processed. The writing mode can include, for example, replacing an original field or adding to a new field, and the like. Based on this, the computing result can be added to the data to be processed according to the writing mode in the configuration file to obtain target data.

[0053] The common request field can be used to indicate a field needed to generate a target request. Based on this, the common request field corresponding to the remote service request can be obtained from the configuration file.

[0054] According to an embodiment of the present disclosure, an asynchronous operator can automatically process remote service requests according to configuration information. A user only needs to flexibly configure the configuration information of the asynchronous operator according to the user's own needs, so that the corresponding downstream operator node can obtain an expected result. The user does not need to understand the complex implementation logic of the business bottom layer.

[0055] According to an embodiment of the present disclosure, the target requests can be classified, for example, and a primary queue and a secondary queue can be set for each type of target request. The primary queue is used to store the requests being sent, and the secondary queue is used to store the requests waiting to be sent. The primary queue and the secondary queue can be message queues.

[0056] After the target request is generated, it can be determined whether the primary queue corresponding to the target request is full. In the case where the primary queue is not full, the target request can be written into the primary queue. In the case where the primary queue is full, the target request can be written into the secondary queue.

[0057] Then, each request in the primary queue can be sent to the computing server respectively. In addition, the requests in the secondary queue can also be migrated to the primary queue periodically. The time interval of migration can be set according to actual needs. The number of requests migrated each time can be determined according to the remaining space in the primary queue.

[0058] According to an embodiment of the present disclosure, the primary queue can be generated based on Redis, for example. The advantage of using Redis for the primary queue is that the overhead of queue creation and destruction is small, and the efficiency of accessing data and viewing the queue length is high.

[0059] According to an embodiment of the present disclosure, the secondary queue can be set in a disk storage medium, for example, and can be generated based on a NoSQL database or a distributed table supporting SCAN (scan), for example. The NoSQL database can include ssdb, for example.

[0060] The method of sending target requests to a computing server shown above will be further described below with reference to Figure 3 and in conjunction with specific embodiments. Those skilled in the art can understand that the following example embodiments are only for understanding the present disclosure, and the present disclosure is not limited thereto.

[0061] Figure 3 A schematic diagram of sending target requests to a computing server according to an embodiment of the present disclosure is schematically shown.

[0062] In Figure 3 , it is shown that in the present embodiment, the secondary queue can include queue 1, queue 2, queue 3, queue 4, …, and the primary queue can include queue 1’, queue 2’, queue 3’, queue 4’, ….

[0063] According to an embodiment of the present disclosure, the target requests to be sent can be classified and written into different queues according to types. For example, the target requests of type A can be written into queue 1 or queue 1', the target requests of type B can be written into queue 2 or queue 2', the target requests of type C can be written into queue 3 or queue 3', and the target requests of type D can be written into queue 4 or queue 4'.

[0064] According to an embodiment of the present disclosure, each primary queue corresponds to a token bucket. For example, queue 1', queue 2', queue 3', queue 4'... correspond to token bucket 1, token bucket 2, token bucket 3, token bucket 4... respectively. The number of tokens in the token bucket can be consistent with the size of the primary queue, so that each request in the primary queue can obtain a token in the corresponding token bucket. The request obtaining the token can be sent to the computing server.

[0065] According to an embodiment of the present disclosure, the data carrying thread is used to periodically migrate the requests in each secondary queue to the corresponding primary queue. For example, the requests in queue 1 can be migrated to queue 1' according to a first time interval, the requests in queue 2 can be migrated to queue 2' according to a second time interval, the requests in queue 3 can be migrated to queue 3' according to a third time interval, and the requests in queue 4 can be migrated to queue 4' according to a fourth time interval. The first time interval, the second time interval, the third time interval and the fourth time interval can be set according to actual needs.

[0066] According to another embodiment of the present disclosure, a request retransmission mechanism can also be set for the target request, so as to retransmit the target request in the case that the computing result is not received, thereby improving the success rate of obtaining the computing result.

[0067] Based on this, Figure 4 The flowchart of the data processing method according to another embodiment of the present disclosure is schematically shown.

[0068] As Figure 4 shown, the data processing method 400 can further include writing, at operation S470, the target request and the retry number corresponding to the target request into the delay message queue.

[0069] According to an embodiment of the present disclosure, the retry number can be pre-set in the configuration file corresponding to the target request. Based on this, the retry number corresponding to the target request can be obtained from the corresponding configuration file.

[0070] Then, in operation S480, it is determined whether the calculation result corresponding to the pending request has been completely received for each pending request in the delay message queue. In a case where it is determined that the calculation result corresponding to the pending request has been completely received, operation S490 is performed. In a case where it is determined that the calculation result corresponding to the pending request has not been completely received, operation S4100 is performed.

[0071] In operation S490, the pending request is deleted from the delay message queue, and the completely received calculation result is added to the pending data to obtain the target data.

[0072] In operation S4100, it is determined whether the retry number corresponding to the pending request is equal to 0. In a case where it is determined that the calculation result corresponding to the pending request has not been completely received and the retry number corresponding to the pending request is greater than 0, operation S4110 is performed. In a case where it is determined that the retry number corresponding to the pending request is equal to 0, operation S4120 is performed.

[0073] According to an embodiment of the present disclosure, for example, the total number of corresponding fields in the remote service request that match the preset field path can be determined as the target number. In a case where the calculation result is received, the target number is subtracted by the number of received calculation results to obtain the number to be received. In a case where it is determined that the number to be received is 0, it can be determined that the calculation result has been completely received. In a case where it is determined that the number to be received is greater than 0, it can be determined that the calculation result has not been completely received.

[0074] According to an embodiment of the present disclosure, the total number of corresponding fields and the received calculation results can be recorded in a cache. The cache can be arranged inside the asynchronous node or outside the asynchronous node, for example, in a storage system outside the asynchronous node. Based on this, the number of received calculation results can be determined based on the total number and the calculation results recorded in the cache.

[0075] According to another embodiment of the present disclosure, the configuration file can further include a forced refresh identifier. The forced refresh identifier is used to indicate whether to force refresh the cache. When the forced refresh identifier indicates to force refresh the cache, the received calculation results recorded in the cache are emptied when the target request is sent.

[0076] In operation S4110, the pending request is sent to the calculation server, and the retry number is reduced by 1. Return to operation S480.

[0077] In operation S4120, the pending request is deleted from the delay message queue, and the received calculation result is added to the pending data to obtain the target data.

[0078] According to an embodiment of the present disclosure, when the number of retries is reduced to 0, it indicates that the number of retries is used up, and the received calculation result can be forced to be issued to the downstream operator node, so that the business data can be guaranteed not to be lost.

[0079] The data processing method shown above will be further described below with reference to specific embodiments. Figure 5 Those skilled in the art can understand that the following example embodiments are only for understanding the present disclosure, and the present disclosure is not limited thereto.

[0080] Exemplarily, in the present embodiment, the asynchronous operator can be provided with a message queue, a common sending module and a common receiving module. The common sending module includes a first-level queue, a second-level queue and a delayed message queue.

[0081] Figure 5 A schematic diagram of a data processing method according to another embodiment of the present disclosure is schematically shown.

[0082] In Figure 5 In operation S501, the asynchronous node loads a configuration file.

[0083] According to an embodiment of the present disclosure, the asynchronous node can receive a remote service request from an upstream operator node. A configuration file corresponding to the remote service request is obtained. The configuration file includes a write-back operator node identifier, a preset field path, a writing mode, a number of retries, a forced cache flushing identifier, and the like, and a common request field. The configuration file may, for example, be a schema.

[0084] In operation S502, the asynchronous node extracts the to-be-processed data and the original path mapping relationship in the remote service request and stores them in a storage system.

[0085] According to an embodiment of the present disclosure, the asynchronous node can extract a corresponding field matching each preset field path in the remote service request as to-be-processed data, and determine an original path of the corresponding field in the remote service request. Then, a mapping relationship between the corresponding field and the original path of the corresponding field, i.e., an original path mapping relationship, is stored in a storage system. The storage system may, for example, include a distributed table storage system.

[0086] Exemplarily, in the present embodiment, the format of the original path mapping relationship can be “field value: original path of the field in the remote service request”. For example:

[0087] {

[0088] “value1”: “data.display.array_a.0.text”,

[0089] " value2 ": " data.display.array_a.1.text ",

[0090] " value3 ": " data.display.news.digest.text ",

[0091] …

[0092] }

[0093] wherein value1, value2 and value3 are field values. The original path of value1 is data.display.array_a.0.text. The original path of value2 is data.display.array_a.1.text. The original path of value3 is data.display.news.digest.text.

[0094] In operation S503, the asynchronous node writes the target request and the corresponding retry number into the message queue.

[0095] According to an embodiment of the present disclosure, the asynchronous node can obtain the common request field in the configuration file. According to the to-be-processed data and the common request field, the target request is generated.

[0096] In operation S504, the common sending module subscribes to the message queue to obtain the target request.

[0097] In operation S505, the common sending module sends the requests in the message queue that need to be sent to the computing server one by one.

[0098] According to an embodiment of the present disclosure, the common sending module determines whether the primary queue is full. In the case where the primary queue is not full, the target request is written into the primary queue. In the case where the primary queue is full, the target request is written into the secondary queue, and the requests in the secondary queue are migrated to the primary queue periodically. Each request in the primary queue is sent in turn.

[0099] In operation S506, the common sending module writes the target request and the corresponding retry number into the delay message queue.

[0100] In operation S507, the computing server performs calculation on the to-be-processed data in the target request to obtain a calculation result, and sends the calculation result to the common receiving module.

[0101] In operation S508, the common receiving module receives the calculation result of the to-be-processed data from the computing server.

[0102] In operation S509, the common receiving module stores the calculation result into the storage system.

[0103] According to an embodiment of the present disclosure, when the calculation result is received, a target field corresponding to the calculation result is determined, and the target field and a path of the target field in the remote service request are recorded in the meta information of the storage system.

[0104] In operation S510, the common receiving module requests to obtain meta information in the storage system. The meta information includes information of the received calculation result.

[0105] In operation S511, the common receiving module determines whether the calculation result required by the remote service request has been completely received according to the meta information. In the case of complete reception, operations S512-S514 are performed. In the case of incomplete reception, operation S515 is performed.

[0106] In operation S512, the common receiving module requests the storage system to obtain all calculation results.

[0107] In operation S513, the storage system returns all calculation results to the common receiving module.

[0108] In operation S514, the common receiving module adds the calculation result to the to-be-processed data according to the writing mode in the configuration file to obtain target data according to the writing mode in the configuration file. The target data is subsequently sent to the corresponding downstream operator node.

[0109] In operation S515, the waiting continues.

[0110] In operation S516, the common sending module periodically reads the request in the delay message queue.

[0111] In operation S517, the common sending module inquires the common receiving module whether the request needs to be retransmitted.

[0112] In the case of retransmission, in operation S518, the common receiving module sends a notification to the common sending module to inform the common sending module that the request needs to be retransmitted.

[0113] For example, the common receiving module compares the target field recorded in the storage system with the original path mapping relationship to determine whether the calculation result corresponding to all target fields has been completely received. In the case of incomplete reception, the common sending module is informed that the request needs to be retransmitted.

[0114] In operation S519, the common sending module updates the number of times of retransmission and writes the updated number of retries into the delay message queue.

[0115] In operation S520, the common sending module retransmits the request.

[0116] In operation S521, the computing server performs calculation on the to-be-processed data in the retransmission request, obtains a calculation result, and sends the calculation result to the common receiving module.

[0117] In operation S522, the common sending module determines whether the retry count has been zeroed. If yes, operation S523 is performed.

[0118] In operation S523, the common sending module notifies the common receiving module to forcibly send the received calculation result to the downstream operator node.

[0119] According to an embodiment of the present disclosure, after receiving the notification, the common receiving module can obtain the currently received calculation result from the storage system, then add the calculation result to the to-be-processed data according to the writing mode in the configuration file to generate target data, and send the target data to the corresponding downstream operator node. Thus, the business data loss is avoided.

[0120] According to an embodiment of the present disclosure, the request for calling the remote service is sent by the common sending module. In the common sending module, unified traffic control can be performed, so that the threat of high instantaneous traffic of the operator node to the stability of the remote service can be reduced.

[0121] The data processing apparatus provided by the present disclosure will be described below. Figure 6 The data processing apparatus provided by the present disclosure will be described below.

[0122] Figure 6 A block diagram of the data processing apparatus according to an embodiment of the present disclosure is schematically shown.

[0123] As shown in Figure 6 The data processing apparatus 600 includes a first receiving module 610, an analysis module 620, a first sending module 630, a second receiving module 640, a generation module 650, and a second sending module 660.

[0124] The first receiving module 610 is configured to receive a remote service request from an upstream operator node.

[0125] The analysis module 620 is configured to analyze the remote service request to obtain to-be-processed data.

[0126] The first sending module 630 is configured to send the to-be-processed data to a computing server.

[0127] The second receiving module 640 is configured to receive a calculation result from the computing server for the to-be-processed data.

[0128] The generation module 650 is configured to generate target data according to the calculation result and the to-be-processed data.

[0129] The second sending module 660 is configured to send the target data to a downstream operator node corresponding to the upstream operator node.

[0130] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0131] Figure 7 A block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown schematically. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0132] As shown in Figure 7 The device 700 includes a computing unit 701 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded into a random access memory (RAM) 703 from a storage unit 708. Various programs and data required for the operation of the device 700 can also be stored in the RAM 703. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0133] Various components in the device 700 are connected to the I / O interface 705, including an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, a speaker, etc.; the storage unit 708, such as a magnetic disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0134] The computing unit 701 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs various methods and processes described above, such as the data processing method. For example, in some embodiments, the data processing method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded onto the RAM 703 and executed by the computing unit 701, one or more steps of the data processing method described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the data processing method by any other suitable means, such as by means of firmware.

[0135] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0136] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0137] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0138] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0139] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0140] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0141] The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in a cloud computing service system to solve the defects of large management difficulty and weak business scalability in traditional physical hosts and VPS services (Virtual Private Server, or simply VPS). The server can also be a server of a distributed system, or a server combined with a blockchain.

[0142] It should be understood that the steps can be reordered, added, or deleted using the various forms of flow shown above. For example, the steps described in the present disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions of the present disclosure can be achieved, which are not limited herein.

[0143] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data processing method, comprising: Receive remote service requests from upstream operator nodes; Parse the remote service request to obtain the data to be processed; The data to be processed is sent to the computing server; Receive the calculation results from the computing server for the data to be processed; Based on the calculation results and the data to be processed, generate target data; as well as The target data is sent to the downstream operator node corresponding to the upstream operator node; The step of sending the data to be processed to the computing server includes: obtaining a common request field corresponding to the remote service request; generating a target request based on the data to be processed and the common request field; and sending the target request to the computing server. The method further includes: writing the target request and the number of retries corresponding to the target request into a delayed message queue; for each pending request in the delayed message queue, determining whether the calculation result corresponding to the pending request has been fully received and whether the number of retries corresponding to the pending request is greater than 0; and if it is determined that the calculation result corresponding to the pending request has not been fully received and the number of retries corresponding to the pending request is greater than 0, sending the pending request to the computing server and decrementing the number of retries by 1. Sending the target request to the computing server includes: writing the target request to the primary queue when the primary queue is not full; writing the target request to a secondary queue when the primary queue is full, and periodically migrating requests from the secondary queue to the primary queue; and sending each request in the primary queue to the computing server; wherein the primary queue is used to store requests that are being sent, and the secondary queue is used to store requests that are waiting to be sent.

2. The method according to claim 1, wherein, The process of parsing the remote service request to obtain the data to be processed includes: Obtain the preset field path corresponding to the remote service request; and The corresponding field in the remote service request that matches the preset field path is identified and used as the data to be processed.

3. The method according to claim 2, further comprising: Determine the total number of corresponding fields in the remote service request that match the preset field path, and use this as the target number; Upon receiving the calculation result Subtract the number of received calculation results from the target number to obtain the number to be received; If the quantity to be received is determined to be 0, it is determined that the calculation result has been fully received; as well as If the quantity to be received is greater than 0, it is determined that the calculation result has not been fully received.

4. The method according to claim 3, wherein, The step of generating target data based on the calculation results and the data to be processed includes: If it is determined that the calculation result is fully received, the fully received calculation result is added to the data to be processed to obtain the target data; and If the number of retries corresponding to the target request is determined to be 0, the received calculation results are added to the data to be processed to obtain the target data.

5. A data processing apparatus, comprising: The first receiving module is used to receive remote service requests from upstream operator nodes; The parsing module is used to parse the remote service request to obtain the data to be processed; The first sending module is used to send the data to be processed to the computing server; The second receiving module is used to receive the calculation results from the computing server for the data to be processed; A generation module is used to generate target data based on the calculation results and the data to be processed; as well as The second sending module is used to send the target data to the downstream operator node corresponding to the upstream operator node; The first sending module is configured to: obtain a common request field corresponding to the remote service request; generate a target request based on the data to be processed and the common request field; and send the target request to the computing server. The first sending module is further configured to: write the target request and the number of retries corresponding to the target request into a delayed message queue; for each pending request in the delayed message queue, determine whether the calculation result corresponding to the pending request has been fully received and whether the number of retries corresponding to the pending request is greater than 0; and if it is determined that the calculation result corresponding to the pending request has not been fully received and the number of retries corresponding to the pending request is greater than 0, send the pending request to the computing server and decrement the number of retries by 1. The first sending module is configured to: write a target request into the primary queue when the primary queue is not full; write the target request into a secondary queue when the primary queue is full, and periodically migrate requests from the secondary queue to the primary queue; and send each request in the primary queue to the computing server; wherein the primary queue is used to store requests that are being sent, and the secondary queue is used to store requests that are waiting to be sent.

6. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.

7. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-4.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-4.

Images