Data processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data processing method, and the method comprises the steps: obtaining a task set and a robot set, the task set comprises a plurality of pieces of task data, and the robot set comprises a plurality of pieces of robot data; multiple pieces of task data in the task set and multiple pieces of robot data in the robot set are combined to obtain multiple scheduling instances, and each scheduling instance comprises one piece of target robot data and at least one piece of target task data; for one scheduling instance, based on one piece of target robot data and at least one piece of target task data, determining total task execution time corresponding to each scheduling instance; based on the total task execution time, target scheduling data are determined in the multiple scheduling instances, and the target scheduling data comprise at least one piece of target robot data and all task data. The working efficiency of the robot can be improved on the whole.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data processing method, device, electronic equipment and storage medium. Background Art

[0002] In the existing data processing process, data docking between different systems is essential. Usually, data docking between different systems is achieved through a fixed set of data receiving / pushing mechanisms. If data from a new system needs to be connected at this time, the developer can only rewrite the code according to the interface document, and the subsequent data interface changes still require the intervention of the developer. When more and more systems are docked, the system development and maintenance costs will increase proportionally. Therefore, an efficient data docking solution is urgently needed to solve these problems. Summary of the invention

[0003] The embodiment of the present invention provides a data processing method, aiming to provide an efficient data docking solution, which can realize dynamic intelligent docking of data with different structures between multiple systems, without the intervention of developers, and reduce the cost of system development and maintenance. The first mapping relationship between the request header data and the receiving end interface is recorded through the first mapping table, so that the receiving end interface and the target unique label can be automatically obtained according to the target request header data, and the second mapping relationship between the request body data and the received data is recorded through the second mapping table, so that the corresponding target received data can be automatically obtained according to the target unique label and the target request body data, thereby realizing dynamic intelligent docking of data with different structures between the push end and the receiving end, and reducing the cost of system development and maintenance.

[0004] In a first aspect, an embodiment of the present invention provides a data processing method, the method comprising the following steps:

[0005] Receive target data to be pushed from a target push end, wherein the target data to be pushed includes target request header data and target request body data;

[0006] Determine a target receiving end interface and a target unique tag in the target request header data based on a first mapping table, wherein the first mapping table includes first mapping relationships between request header data and receiving end interfaces, and each first mapping relationship corresponds to a unique tag;

[0007] Based on the target unique tag, determining the target received data corresponding to the target request body data in a second mapping table, wherein the second mapping table includes second mapping relationships between the request body data and the received data, and each second mapping relationship corresponds to one of the unique tags;

[0008] Based on the target receiving end interface, the target receiving data is pushed to the target receiving end.

[0009] Optionally, before determining, in the second mapping table based on the target unique tag, the target received data corresponding to the target request body data, the method further includes:

[0010] Acquire at least one request body data and at least one received data, wherein the request body data includes a first field name, a first field type, and a first field, wherein the first field name is the naming data of the first field, and the first field type is the type data of the first field, and the received data includes a second field name, a second field type, and a second field, wherein the second field name is the naming data of the second field, and the second field type is the type data of the second field;

[0011] Determine a third mapping relationship between the first field and the second field based on the first field name, the first field type, the second field name, and the second field type;

[0012] Determine a second mapping relationship between the request body data and the received data based on a third mapping relationship between the first field and the second field;

[0013] Based on the second mapping relationship between the request body data and the received data, a second mapping table is constructed.

[0014] Optionally, determining a third mapping relationship between the first field and the second field based on the first field name, the first field type, the second field name, and the second field type includes:

[0015] Determine whether the first field name and the first field type are the same as the second field name and the second field type, respectively;

[0016] If the first field name and the first field type are respectively the same as the second field name and the second field type, determining that there is a third mapping relationship between the first field and the second field;

[0017] If the first field name and the first field type are different from the second field name and the second field type respectively, a third mapping relationship between the first field and the second field is determined based on the first field name and the second field name.

[0018] Optionally, determining a third mapping relationship between the first field and the second field based on the first field name and the second field name includes:

[0019] Determine whether the first field name is the same as or similar to the second field name;

[0020] If the first field name is the same as or similar to the second field name, determining that there is a third mapping relationship between the first field and the second field;

[0021] If the first field name is not the same as or similar to the second field name, a third mapping relationship between the first field and the second field is determined based on the semantics of the first field name and the second field name.

[0022] Optionally, constructing a second mapping table based on a second mapping relationship between the request body data and the received data includes:

[0023] Create an initial mapping table;

[0024] Acquire sample push data, where the sample push data includes sample request body data;

[0025] Based on the second mapping relationship and the initial mapping table, the sample receiving data corresponding to the sample pushing data is pushed to the sample receiving end;

[0026] Receiving response data from the sample receiving end, and updating and debugging the initial mapping table based on the response data;

[0027] The second mapping table is obtained through updating and debugging for a preset number of times.

[0028] Optionally, updating and debugging the initial mapping table based on the response data includes:

[0029] Parsing the response data to determine whether there is an error in the response data;

[0030] If there is no error in the response data, updating the initial mapping table based on the response data;

[0031] If there is an error in the response data, determining the error type based on the response data;

[0032] The second mapping relationship corresponding to the response data in the initial mapping table is debugged based on the error type until there is no error in the response data.

[0033] Optionally, before determining the target receiving end interface and the target unique tag in the target request header data based on the first mapping table, the method further includes:

[0034] In the first mapping table, a unique label is determined through the request header data and the receiving end interface;

[0035] In the first mapping table, the unique tag is associated with the first mapping relationship.

[0036] In a second aspect, an embodiment of the present invention further provides a data processing device, the data processing device comprising:

[0037] A receiving module, used for receiving target data to be pushed from a target push end, wherein the target data to be pushed includes target request header data and target request body data;

[0038] A first processing module, configured to determine a target receiving end interface and a target unique tag in the target request header data based on a first mapping table, wherein the first mapping table includes a first mapping relationship between the request header data and the receiving end interface, each first mapping relationship corresponding to a unique tag;

[0039] A second processing module, configured to determine, based on the target unique tag, in a second mapping table, target received data corresponding to the target request body data, wherein the second mapping table includes second mapping relationships between the request body data and the received data, and each second mapping relationship corresponds to one of the unique tags;

[0040] The push module is used to push the target receiving data to the target receiving end based on the target receiving end interface.

[0041] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data processing method provided in the embodiment of the present invention when executing the computer program.

[0042] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the data processing method provided in the embodiment of the invention are implemented.

[0043] In an embodiment of the present invention, the target data to be pushed of the target push end is received, and the target data to be pushed includes the target request header data and the target request body data; based on the first mapping table, the target receiving end interface and the target unique tag are determined in the target request header data, and the first mapping table includes the first mapping relationship between the request header data and the receiving end interface, and each first mapping relationship corresponds to a unique tag; based on the target unique tag, the target receiving data corresponding to the target request body data is determined in the second mapping table, and the second mapping table includes the second mapping relationship between the request body data and the receiving data, and each second mapping relationship corresponds to a unique tag; based on the target receiving end interface, the target receiving data is pushed to the target receiving end. The first mapping relationship between the request header data and the receiving end interface is recorded in the first mapping table, so that the receiving end interface and the target unique tag can be automatically obtained according to the target request header data, and the second mapping relationship between the request body data and the receiving data is recorded in the second mapping table, so that the corresponding target receiving data can be automatically obtained according to the target unique tag and the target request body data, thereby realizing the dynamic intelligent docking of data with different structures between the push end and the receiving end, and reducing the system development and maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 is a flow chart of a data processing method provided by an embodiment of the present invention;

[0046] Figure 2 is a flow chart of a data processing method provided by an embodiment of the present invention;

[0047] Figure 3 is a schematic diagram of constructing a second mapping relationship provided by an embodiment of the present invention;

[0048] Figure 4 is a schematic diagram of debugging a second mapping table provided in an embodiment of the present invention;

[0049] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] like Figure 1 As shown, Figure 1 : is a flow chart of a data processing method provided by an embodiment of the present invention, the data processing method comprising the steps of:

[0052] 101. Receive target data to be pushed from a target push end.

[0053] In an embodiment of the present invention, the above-mentioned data processing method is applied to a data docking platform or a data docking plug-in. The above-mentioned data docking platform can be constructed by a server or a distributed server. The above-mentioned data docking platform includes a data processing application and interfaces of multiple data systems. The data of each data system can be obtained through the interface of the data system. The data of different data systems have different data structures. The data docking platform converts the received push-end data into received data that conforms to the data structure of the receiving end through the data processing application, and pushes the received data to the receiving end. Of course, the above-mentioned data docking plug-in is embedded with a data processing application. The data docking plug-in relies on the main device to obtain data from different data systems and push the received data to the receiving end. The docking plug-in is only responsible for converting the push-end data into received data that conforms to the data structure of the receiving end.

[0054] The target data to be pushed includes target request header data and target request body data.

[0055] The above request header data may include DATA_TYPE, where DATA_TYPE is used to identify the type of data accessed, such as accessing event data: DATA_TYPE=event.

[0056] The request body data can include fieldInteger, fieldString, and fieldDouble. fieldInteger represents an integer data field, fieldString represents a string data field, and fieldDouble represents a double-precision floating point data field. The input format of the request body data can be in standard JSON format. The following is a simple example:

[0057]

[0058] The target push end is an electronic device equipped with any one of the multiple data systems, and the target push end is used to push the data in the data system to the target receiving end, and the target receiving end and the target push end are electronic devices equipped with different data systems. The above target push end and the above target receiving end both have their own corresponding IP (Internet Protocol) addresses.

[0059] 102. Based on the first mapping table, determine the target receiving end interface and the target unique tag in the target request header data.

[0060] In an embodiment of the present invention, after obtaining the target data to be pushed, the target data to be pushed can be parsed to obtain the request header data and request body data of the target data to be pushed, and the request header data of the target data to be pushed can be determined as the target request header data, and the request body data of the target device can be determined as the target request body data.

[0061] Maintain a first mapping table to determine which data system the data to be pushed comes from and which data system it needs to be pushed to. The first mapping table includes a first mapping relationship between the request header data and the receiving end interface, and each first mapping relationship corresponds to a unique label. Specifically, the first mapping table includes multiple first mapping data, each of which corresponds to a first mapping relationship, and each of which corresponds to a unique identifier. The first mapping data mainly includes the primary key unique id (id), the push end IP (source_ip), the access data type (data_type), the receiving end interface (target_url), and the unique identifier (label). The above unique identifier can also be called a label. The above first mapping table can be shown in Table 1:

[0062] Table 1

[0063]

[0064]

[0065] The first mapping table can be maintained manually and is the basis for realizing data connection with the NN system. It can be maintained in full before the second mapping table is constructed. The "unique identifier" field must be guaranteed to be unique. The unique identifier can be generated based on a digest algorithm such as MD5. Specifically, it can be generated using the request header data through the digest algorithm MD5.

[0066] In the first mapping table, the corresponding first mapping data can be determined by the push-end IP and the access data type, and the target receiving end interface and the target unique identifier can be determined in the corresponding first mapping data.

[0067] 103. Based on the target unique tag, determine the target received data corresponding to the target request body data in the second mapping table.

[0068] In an embodiment of the present invention, the second mapping table includes a second mapping relationship between the request body data and the received data, and each second mapping relationship corresponds to a unique label. Specifically, the second mapping table includes multiple second mapping data, each second mapping data corresponds to a second mapping relationship, and each second mapping data corresponds to a unique identifier. The second mapping data mainly includes the primary key unique id (id), the pusher field (source_field), the pusher field name (source_field_name), the pusher field type (source_field_type), the receiver field (target_field), the receiver field name (target_field_name), the receiver field type (source_field_type), and a unique identifier (label). The above unique identifier can also be called a label. The above second mapping table can be shown in Table 2:

[0069] Table 2

[0070]

[0071] The second mapping table is the core of realizing dynamic docking of NN system data, in which each second mapping data records the mapping relationship of a field between two different data systems. The unique identifier label is the unique identification field in the first mapping table, which is used to identify that the second mapping data belongs to the corresponding receiving port in the corresponding first mapping data.

[0072] Through the second mapping table, the target pusher field in the target request body can be mapped to the receiver field required by the receiver, and the receiver field can be used as the target reception data. In a possible embodiment, the receiver field, the receiver field name and the receiver field type can be packaged to obtain the target reception data. The above-mentioned target reception data can be understood as data to be received by the target receiving end.

[0073] 104. Based on the target receiving end interface, the target receiving data is pushed to the target receiving end.

[0074] In an embodiment of the present invention, after obtaining the target receiving data, the target receiving data can be pushed to the target receiving end through the target receiving end interface. Since the target receiving data passes through the second mapping table, the original target data to be pushed is converted into the data structure of the target receiving end, so that the target receiving end can receive the target receiving data that conforms to the data structure.

[0075] In an embodiment of the present invention, the target data to be pushed of the target push end is received, and the target data to be pushed includes the target request header data and the target request body data; based on the first mapping table, the target receiving end interface and the target unique tag are determined in the target request header data, and the first mapping table includes the first mapping relationship between the request header data and the receiving end interface, and each first mapping relationship corresponds to a unique tag; based on the target unique tag, the target receiving data corresponding to the target request body data is determined in the second mapping table, and the second mapping table includes the second mapping relationship between the request body data and the receiving data, and each second mapping relationship corresponds to a unique tag; based on the target receiving end interface, the target receiving data is pushed to the target receiving end. The first mapping relationship between the request header data and the receiving end interface is recorded in the first mapping table, so that the receiving end interface and the target unique tag can be automatically obtained according to the target request header data, and the second mapping relationship between the request body data and the receiving data is recorded in the second mapping table, so that the corresponding target receiving data can be automatically obtained according to the target unique tag and the target request body data, thereby realizing the dynamic intelligent docking of data with different structures between the push end and the receiving end, and reducing the system development and maintenance costs.

[0076] It can be understood that in the specific implementation of this application, it involves push data, reception data, response data, communication data and other related data. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data, as well as the training, deployment and calling of large models, must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0077] Optionally, before determining the target received data corresponding to the target request body data in the second mapping table based on the target unique tag, at least one request body data and at least one received data can also be obtained, the request body data including a first field name, a first field type and a first field, the first field name is the naming data of the first field, and the first field type is the type data of the first field, the received data including a second field name, a second field type and a second field, the second field name is the naming data of the second field, and the second field type is the type data of the second field; based on the first field name, the first field type, the second field name, and the second field type, determine the third mapping relationship between the first field and the second field; based on the third mapping relationship between the first field and the second field, determine the second mapping relationship between the request body data and the received data; based on the second mapping relationship between the request body data and the received data, construct a second mapping table.

[0078] In an embodiment of the present invention, the name of the first field may be the pusher field, the type of the first field may be the pusher field type, the first field may be the pusher field, the second field name may be the receiver field name, the second field type may be the receiver field type, and the second field may be the receiver field. In a possible embodiment, the name of the second field may be the pusher field, the type of the second field may be the pusher field type, the second field may be the pusher field, the name of the first field may be the receiver field name, the first field type may be the receiver field type, and the first field may be the receiver field.

[0079] The request body data may be one or more, and the received data may be one or more. In the process of constructing the second mapping table, a request body data set and a received data set may be constructed, wherein the request body data set includes at least one request body data, and the received data set includes at least one received data set.

[0080] The second mapping table is used to map the first field to the second field, or to map the second field to the first field, so that the target receiving end can receive field data with a consistent data structure. Therefore, the second mapping table includes a third mapping relationship between the first field and the second field.

[0081] Whether the third mapping relationship between the first field and the second field is established can be determined based on whether the first field name is the same as the second field name, and whether the first field type is the same as the second field type. If the first field name is the same as the second field name, and the first field type is the same as the second field type, it can be determined that the second mapping relationship between the first field and the second field is established, that is, it is determined that the third mapping relationship exists between the first field and the second field. Otherwise, it can be determined that the second mapping relationship between the first field and the second field is not established, that is, it is determined that the third mapping relationship does not exist between the first field and the second field.

[0082] In the case where there is a third mapping relationship between the first field and the second field, it can be directly determined that the second mapping relationship between the request body data and the received data holds, that is, there is a second mapping relationship between the request body data and the received data.

[0083] Optionally, in the step of determining the third mapping relationship between the first field and the second field based on the first field name, the first field type, the second field name, and the second field type, it can be determined whether the first field name and the first field type are respectively the same as the second field name and the second field type; if the first field name and the first field type are respectively the same as the second field name and the second field type, it is determined that there is a third mapping relationship between the first field and the second field; if the first field name and the first field type are respectively different from the second field name and the second field type, the third mapping relationship between the first field and the second field is determined based on the first field name and the second field name.

[0084] In the embodiment of the present invention, it can be further determined whether the first field name is the same as the second field name, and whether the first field type is the same as the second field type. If they are the same, it means that the first field and the second field are different data structures of the same field, and it can be determined that the third mapping relationship between the first field and the second field is established, that is, there is a third mapping relationship between the first field and the second field. If they are not the same, only the first field name and the second field name can be considered, and based on the first field name and the second field name, the third mapping relationship between the first field and the second field can be determined.

[0085] For example, if Figure 2 As shown, the third mapping relationship between the data pusher field and the receiver field is established through mapping rules. Assuming that the field currently being processed is the pusher field A, then set S is the receiver field. Conversely, if field A is the receiver field, then set S is the pusher field. The mapping rule can be: preferentially filter out fields in set S whose field names and field types are consistent with field A. If there is a field in set S whose field name and field type are consistent with field A, it is determined that the field is mappable with field A, that is, it is determined that there is a third mapping relationship between the field and field A. Otherwise, in set S, filter out fields that can be mapped with field A by field name.

[0086] Optionally, in the step of determining the third mapping relationship between the first field and the second field based on the first field name and the second field name, it can be determined whether the first field name and the second field name are the same; if the first field name and the second field name are the same or similar, it is determined that there is a third mapping relationship between the first field and the second field; if the first field name and the second field name are not the same or similar, the third mapping relationship between the first field and the second field is determined based on the semantics of the first field name and the second field name.

[0087] In an embodiment of the present invention, when one of the comparisons between the first field name and the second field name, and between the first field type and the second field type is different, if it is further determined whether the first field name and the second field name are the same or similar, if the first field name and the second field name are the same or similar, it means that the first field and the second field are different data structures of the same field, then it can be determined that the third mapping relationship between the first field and the second field is established, that is, there is a third mapping relationship between the first field and the second field.

[0088] In a possible embodiment, in the step of determining whether the first field name is the same or similar to the second field name, it can be determined whether the first field name is the same as the second field name. If the first field name is not the same as the second field name, it is then determined whether the first field name is similar to the second field name. In the process of determining whether the first field name is similar to the second field name, the historical field name of the first field and the historical field name of the second field can be obtained. The above historical field names can be understood as field names under different versions of the corresponding data system. The first field name and the historical field name of the first field are added to the first set, and the second field name and the historical field name of the second field are added to the second set. In the first set, the field names are sorted from large to small according to the character length of each field name. Similarly, in the second set, the field names are sorted from large to small according to the character length of each field name. For multiple different field names with the same character length, the field names are sorted from large to small according to the number of reuses. For multiple different field names with the same number of reuses, the vector distance between the word vector of the field name and the word vector of the corresponding field can be calculated, and the vector distance can be sorted from small to large. The above word vector can be obtained by a word embedding algorithm. After obtaining the sorted first set and the second set, the first similarity between the first set and the second set can be calculated. If the first similarity is greater than or equal to the preset first similarity threshold, it can be determined that the first field name is similar to the second field name. Alternatively, the field names are sampled at a preset interval in the first set to obtain a first field name sequence, and the field names are also sampled at a preset interval in the second set to obtain a second field name sequence, and the second similarity between the first field name sequence and the second field name sequence is calculated. If the second similarity is greater than or equal to the preset second similarity threshold, it can be determined that the first field name is similar to the second field name. The above interval can be an arithmetic interval or a Fibonacci sequence interval. The first field name sequence and the second field name sequence obtained by Fibonacci sequence interval sampling can be more representative, so that the result of whether the first field name is similar to the second field name is more reliable.

[0089] If the first field name is not the same as or similar to the second field name, the semantics between the first field name and the second field name can be extracted respectively to determine whether the semantics between the first field name and the second field name are similar. If the semantics between the first field name and the second field name are similar, it means that the first field and the second field are different data structures of the same field, and it can be determined that the third mapping relationship between the first field and the second field is established, that is, there is a third mapping relationship between the first field and the second field. If the semantics between the first field name and the second field name are not similar, it can be determined that the third mapping relationship between the first field and the second field is not established, that is, there is no third mapping relationship between the first field and the second field.

[0090] Specifically, in the process of extracting the semantics between the first field name and the second field name, we can rely on NLP (natural language processing) semantic analysis capabilities to determine whether the first field name and the second field name are semantically similar, thereby filtering out fields in set S whose field names are semantically similar to the field name of field A.

[0091] In a possible embodiment, when the first field name and the second field name are in different languages, the translation capability of the customizable vocabulary can also be relied upon to determine whether the first field name and the second field name are semantically similar after being translated into Chinese, and then filter the fields in set S whose field names are semantically similar to the field name of field A.

[0092] After determining the third mapping relationship between the first field and the second field, it can be determined that the corresponding request body data and received data have a second mapping relationship, and then the request body data and received data with the second mapping relationship are added to a mapping table to construct a second mapping table.

[0093] Please continue to refer to Figure 2 ,exist Figure 2In the above, first determine whether there is a field in set S whose field name and field type are consistent with field A. If so, it can be mapped, otherwise the next rule is executed; determine whether there is a field in set S whose field name is consistent with field A. If so, it can be mapped, otherwise the next rule is executed; determine whether there is a field in set S whose field name is consistent with / similar to field A. Here, you can rely on the NLP semantic analysis capability to determine whether the two field names are semantically consistent or similar. If so, they can be mapped, otherwise the next rule is executed; rely on the NLP semantic analysis capability to determine whether the two field names are semantically similar, and determine whether there is a field in set S whose field name and field A name meet the rule. If so, they can be mapped, otherwise the next rule is executed; rely on the translation capability of the customizable vocabulary to determine whether the two field names are semantically similar after translation into Chinese. Whether there is a field in set S whose field name and field A name meet the rule. If so, they can be mapped, otherwise the end.

[0094] The second mapping table may be constructed manually, in batches of documents, or automatically.

[0095] The above-mentioned manual construction can be constructed based on a web page, providing a web page function to support manual maintenance of a second mapping relationship, including a pusher field, a pusher field name, a pusher field type, a receiver field, a receiver field name, a receiver field type, and a label to which the second mapping relationship belongs.

[0096] The above-mentioned document batch construction can be constructed based on the batch import of web pages, providing web page function to import data pusher and receiver API interface documents (word, markdown, openapi and other formats), and obtaining the field information of both parties after parsing the document. The field information of both parties includes the pusher field, the pusher field name, the pusher field type, the receiver field, the receiver field name, and the receiver field type. After traversing the pusher field and processing it according to the above-mentioned mapping rules, a third mapping relationship is established with the receiver field; and then the pusher field, the pusher field name, and the pusher field type are used as request body data, and the receiver field, the receiver field name, and the receiver field type are used as received data to establish a second mapping relationship between the request body data and the received data.

[0097] The automatic construction can be implemented by constructing a debugging and optimization module, which can complete the construction and dynamic optimization of the second mapping table without manual intervention. The debugging and optimization module can perform debugging and optimization by pushing data and responding to data through multiple debugging, and after the debugging and optimization is completed, a usable second mapping table is obtained.

[0098] Optionally, in the step of constructing a second mapping table based on a second mapping relationship between the request body data and the received data, an initial mapping table can be created; sample push data is obtained, and the sample push data includes sample request body data; based on the second mapping relationship and the initial mapping table, the sample received data corresponding to the sample push data is pushed to the sample receiving end; the response data from the sample receiving end is received, and the initial mapping table is updated and debugged based on the response data; and the second mapping table is obtained through a preset number of updates and debugging.

[0099] In an embodiment of the present invention, the sample push data is used for debugging and optimizing the process. During the debugging and optimizing process, some sample push data may be collected or constructed. The sample push data may correspond to a sample push end. The sample push end may be understood as a push end that generates sample push data. The data structure of the sample push data is consistent with the data structure of the sample push end. Further, the sample push data may be push data corresponding to a field that actually needs to be pushed. The sample receiving end is a receiving end that actually needs to receive a field.

[0100] The sample push data includes sample request header data and sample request body data. The data structure of the sample request header data is the same as the data structure of the request header data, that is, the sample request header data may include DATA_TYPE. The sample request body data may include a pusher field (source_field), a pusher field name (source_field_name), and a pusher field type (source_field_type), wherein the pusher field may include fieldInteger, fieldString, and fieldDouble.

[0101] Specifically, for a piece of sample push data, the corresponding second mapping relationship can be constructed through multiple debugging. In the first debugging, because the second mapping table is empty, the sample push data and response data are:

[0102] Sample push data (empty):

[0103]

[0104]

[0105] After debugging, the second mapping table is updated as follows:

[0106] id source_field source_field_name source_field_type target_field target_field_name source_field_type label 1 fieldInteger Numeric type fields Integer fieldInteger1 Numeric type fields Integer label

[0107] In the second debugging, the sample push data and response data are: Push data:

[0108]

[0109] After debugging, the second mapping table is updated as follows:

[0110] id source_field source_field_name source_field_type target_field target_field_name source_field_type label 1 fieldInteger Numeric type fields Integer fieldInteger1 Numeric type fields Integer label 2 fieldString String type field String fieldString1 String type field String label

[0111] In the third debugging, the sample push data and response data are:

[0112] Sample push data:

[0113]

[0114] After debugging, the second mapping table is updated as follows:

[0115] id source_field source_field_name source_field_type target_field target_field_name source_field_type label 1 fieldInteger Numeric type fields Integer fieldInteger1 Numeric type fields Integer label 2 fieldString String type field String fieldString1 String type field String label 3 fieldDouble Floating point type fields Double fieldDouble1 Floating point type fields Double label

[0116] After several debuggings, the mapping table is built and normal data connection can be carried out subsequently.

[0117] The debugging optimization module is used to perform multiple debuggings, and the debugging is performed by pushing data and responding to data. After the debugging is completed, a usable second mapping table is obtained.

[0118] Optionally, in the step of updating and debugging the initial mapping table based on the response data, the response data can be parsed to determine whether there are errors in the response data; if there are no errors in the response data, the initial mapping table is updated based on the response data; if there are errors in the response data, the error type is determined based on the response data; based on the error type, the second mapping relationship corresponding to the response data in the initial mapping table is debugged until there are no errors in the response data.

[0119] In the embodiment of the present invention, the above error types may include field missing, field type error, and field redundancy.

[0120] Please refer to Figure 3 ,exist Figure 3 In the data access of the pusher, it depends on the data access module, and it can be accessed according to the data format in the data access module; the data access module is used to pass the pusher data in.

[0121] Relying on the first mapping table, the receiving party interface and label are determined through the pusher IP and DATA_TYPE in the request header;

[0122] Relying on the second mapping table, searching for field mapping records according to the determined tags, and relying on the push module to build the data structure required by the receiver;

[0123] If the second mapping table is not constructed in advance, that is, only the initial mapping table exists, the pushed data structure is an empty JSON;

[0124] Depending on the recipient's response specification, parse the response body:

[0125] If the error type is field missing, the field mapping relationship is established based on the mapping rule. In this case, field A is the missing field, and set S is all the fields of the pusher.

[0126] If the error type is a field type error, the mapping table field type will be updated;

[0127] If the error type is field redundancy, the corresponding mapping record in the mapping table will be deleted;

[0128] Continue with the next round of debugging until the receiver responds correctly;

[0129] After the receiving party responds correctly, the final second mapping table can be obtained, and the normal data docking process is carried out through the second mapping table that has been debugged and optimized.

[0130] Optionally, before determining the target receiving end interface and the target unique tag in the target request header data based on the first mapping table, the unique tag can also be determined through the request header data and the receiving end interface in the first mapping table; in the first mapping table, the unique tag is associated with the first mapping relationship.

[0131] In an embodiment of the present invention, the request header data may include source_ip, data_type, and target_url. The above unique identification field must be guaranteed to be unique, and the unique identification may be generated based on a digest algorithm such as MD5, and specifically may be generated using the three fields of source_ip, data_type, and target_url through the digest algorithm MD5.

[0132] After the unique identifier is generated, the unique identifier is added to the first mapping relationship corresponding to the request header data in the first mapping table, and is also added to the second mapping relationship corresponding to the request body data in the second mapping table.

[0133] like Figure 4 As shown, an embodiment of the present invention provides a data processing device, the data processing device comprising:

[0134] The receiving module 401 is used to receive target data to be pushed from a target push end, wherein the target data to be pushed includes target request header data and target request body data;

[0135] A first processing module 402 is used to determine a target receiving end interface and a target unique tag in the target request header data based on a first mapping table, wherein the first mapping table includes a first mapping relationship between the request header data and the receiving end interface, and each first mapping relationship corresponds to a unique tag;

[0136] A second processing module 403 is used to determine, based on the target unique tag, the target received data corresponding to the target request body data in a second mapping table, wherein the second mapping table includes second mapping relationships between the request body data and the received data, and each second mapping relationship corresponds to one of the unique tags;

[0137] The push module 404 is used to push the target receiving data to the target receiving end based on the target receiving end interface.

[0138] Optionally, the device further comprises:

[0139] an acquisition module, configured to acquire at least one request body data and at least one received data, wherein the request body data includes a first field name, a first field type, and a first field, wherein the first field name is the naming data of the first field, and the first field type is the type data of the first field, and the received data includes a second field name, a second field type, and a second field, wherein the second field name is the naming data of the second field, and the second field type is the type data of the second field;

[0140] A third processing module, configured to determine a third mapping relationship between the first field and the second field based on the first field name, the first field type, the second field name, and the second field type;

[0141] a fourth processing module, configured to determine a second mapping relationship between the request body data and the received data based on a third mapping relationship between the first field and the second field;

[0142] A construction module is used to construct a second mapping table based on a second mapping relationship between the request body data and the received data.

[0143] Optionally, the third processing module is also used to determine whether the first field name and the first field type are respectively the same as the second field name and the second field type; if the first field name and the first field type are respectively the same as the second field name and the second field type, then it is determined that there is a third mapping relationship between the first field and the second field; if the first field name and the first field type are respectively different from the second field name and the second field type, then based on the first field name and the second field name, the third mapping relationship between the first field and the second field is determined.

[0144] Optionally, the third processing module is also used to determine whether the first field name is the same as or similar to the second field name; if the first field name is the same as or similar to the second field name, it is determined that there is a third mapping relationship between the first field and the second field; if the first field name is not the same as or similar to the second field name, the third mapping relationship between the first field and the second field is determined based on the semantics of the first field name and the second field name.

[0145] Optionally, the construction module is also used to create an initial mapping table; obtain sample push data, the sample push data including sample request body data; based on the second mapping relationship and the initial mapping table, push the sample reception data corresponding to the sample push data to the sample receiving end; receive response data from the sample receiving end, and update and debug the initial mapping table based on the response data; obtain the second mapping table through a preset number of update and debugging.

[0146] Optionally, the construction module is also used to parse the response data to determine whether there are any errors in the response data; if there are no errors in the response data, the initial mapping table is updated based on the response data; if there are errors in the response data, the error type is determined based on the response data; based on the error type, the second mapping relationship corresponding to the response data in the initial mapping table is debugged until there are no errors in the response data.

[0147] Optionally, the device further comprises:

[0148] a fifth processing module, configured to determine a unique tag in the first mapping table through the request header data and the receiving end interface;

[0149] An associating module is used to associate the unique tag with the first mapping relationship in the first mapping table.

[0150] like Figure 5 As shown, an embodiment of the present invention further provides an electronic device, including a processor, and the processor can execute any one of the above data processing methods.

[0151] Specifically, it includes a processor 501 and a memory 502, and a computer program for executing the data processing method which is stored in the memory 502 and can be run on the processor 501, wherein:

[0152] The processor 501 runs the computer program of the data processing method stored in the memory 502 to perform the following steps:

[0153] Receive target data to be pushed from a target push end, wherein the target data to be pushed includes target request header data and target request body data;

[0154] Determine a target receiving end interface and a target unique tag in the target request header data based on a first mapping table, wherein the first mapping table includes first mapping relationships between request header data and receiving end interfaces, and each first mapping relationship corresponds to a unique tag;

[0155] Based on the target unique tag, determining the target received data corresponding to the target request body data in a second mapping table, wherein the second mapping table includes second mapping relationships between the request body data and the received data, and each second mapping relationship corresponds to one of the unique tags;

[0156] Based on the target receiving end interface, the target receiving data is pushed to the target receiving end.

[0157] Optionally, before determining, in the second mapping table, the target received data corresponding to the target request body data based on the target unique tag, the method executed by the processor 501 further includes:

[0158] Acquire at least one request body data and at least one received data, wherein the request body data includes a first field name, a first field type, and a first field, wherein the first field name is the naming data of the first field, and the first field type is the type data of the first field, and the received data includes a second field name, a second field type, and a second field, wherein the second field name is the naming data of the second field, and the second field type is the type data of the second field;

[0159] Determine a third mapping relationship between the first field and the second field based on the first field name, the first field type, the second field name, and the second field type;

[0160] Determine a second mapping relationship between the request body data and the received data based on a third mapping relationship between the first field and the second field;

[0161] Based on the second mapping relationship between the request body data and the received data, a second mapping table is constructed.

[0162] Optionally, the determining, by the processor 501, of a third mapping relationship between the first field and the second field based on the first field name, the first field type, the second field name, and the second field type includes:

[0163] Determine whether the first field name and the first field type are the same as the second field name and the second field type, respectively;

[0164] If the first field name and the first field type are respectively the same as the second field name and the second field type, determining that there is a third mapping relationship between the first field and the second field;

[0165] If the first field name and the first field type are different from the second field name and the second field type respectively, a third mapping relationship between the first field and the second field is determined based on the first field name and the second field name.

[0166] Optionally, the determining, by the processor 501, a third mapping relationship between the first field and the second field based on the first field name and the second field name includes:

[0167] Determine whether the first field name is the same as or similar to the second field name;

[0168] If the first field name is the same as or similar to the second field name, determining that there is a third mapping relationship between the first field and the second field;

[0169] If the first field name is not the same as or similar to the second field name, a third mapping relationship between the first field and the second field is determined based on the semantics of the first field name and the second field name.

[0170] Optionally, the constructing of a second mapping table based on a second mapping relationship between the request body data and the received data performed by the processor 501 includes:

[0171] Create an initial mapping table;

[0172] Acquire sample push data, where the sample push data includes sample request body data;

[0173] Based on the second mapping relationship and the initial mapping table, the sample receiving data corresponding to the sample pushing data is pushed to the sample receiving end;

[0174] Receiving response data from the sample receiving end, and updating and debugging the initial mapping table based on the response data;

[0175] The second mapping table is obtained through updating and debugging for a preset number of times.

[0176] Optionally, the updating and debugging of the initial mapping table based on the response data performed by the processor 501 includes:

[0177] Parsing the response data to determine whether there is an error in the response data;

[0178] If there is no error in the response data, updating the initial mapping table based on the response data;

[0179] If there is an error in the response data, determining the error type based on the response data;

[0180] The second mapping relationship corresponding to the response data in the initial mapping table is debugged based on the error type until there is no error in the response data.

[0181] Optionally, before determining the target receiving end interface and the target unique tag in the target request header data based on the first mapping table, the method executed by the processor 501 further includes:

[0182] In the first mapping table, a unique label is determined through the request header data and the receiving end interface;

[0183] In the first mapping table, the unique tag is associated with the first mapping relationship.

[0184] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the computer program implements the various processes of the data processing method or the application-side data processing method provided by the embodiment of the present invention, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0185] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0186] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises the following steps: Receive target data to be pushed from a target push end, wherein the target data to be pushed includes target request header data and target request body data; Determine a target receiving end interface and a target unique tag in the target request header data based on a first mapping table, wherein the first mapping table includes a first mapping relationship between the request header data and the receiving end interface, and each first mapping relationship corresponds to a unique tag; Based on the target unique tag, determining the target received data corresponding to the target request body data in a second mapping table, wherein the second mapping table includes second mapping relationships between the request body data and the received data, and each second mapping relationship corresponds to one of the unique tags; Based on the target receiving end interface, the target receiving data is pushed to the target receiving end.

2. The data processing method according to claim 1, characterized in that: Before determining, in the second mapping table, the target received data corresponding to the target request body data based on the target unique tag, the method further includes: Acquire at least one request body data and at least one received data, wherein the request body data includes a first field name, a first field type, and a first field, wherein the first field name is the naming data of the first field, and the first field type is the type data of the first field, and the received data includes a second field name, a second field type, and a second field, wherein the second field name is the naming data of the second field, and the second field type is the type data of the second field; Determine a third mapping relationship between the first field and the second field based on the first field name, the first field type, the second field name, and the second field type; Determine a second mapping relationship between the request body data and the received data based on a third mapping relationship between the first field and the second field; Based on the second mapping relationship between the request body data and the received data, a second mapping table is constructed.

3. The data processing method according to claim 2, characterized in that: The determining, based on the first field name, the first field type, the second field name, and the second field type, a third mapping relationship between the first field and the second field includes: Determine whether the first field name and the first field type are the same as the second field name and the second field type, respectively; If the first field name and the first field type are respectively the same as the second field name and the second field type, determining that there is a third mapping relationship between the first field and the second field; If the first field name and the first field type are different from the second field name and the second field type, respectively, a third mapping relationship between the first field and the second field is determined based on the first field name and the second field name.

4. The data processing method according to claim 3, characterized in that: The determining, based on the first field name and the second field name, a third mapping relationship between the first field and the second field includes: Determine whether the first field name is the same as or similar to the second field name; If the first field name is the same as or similar to the second field name, determining that there is a third mapping relationship between the first field and the second field; If the first field name is not the same as or similar to the second field name, a third mapping relationship between the first field and the second field is determined based on the semantics of the first field name and the second field name.

5. The data processing method according to any one of claims 2 to 4, characterized in that: The constructing a second mapping table based on a second mapping relationship between the request body data and the received data includes: Create the initial mapping table; Acquire sample push data, where the sample push data includes sample request body data; Based on the second mapping relationship and the initial mapping table, the sample receiving data corresponding to the sample pushing data is pushed to the sample receiving end; Receiving response data from the sample receiving end, and updating and debugging the initial mapping table based on the response data; The second mapping table is obtained through updating and debugging for a preset number of times.

6. The data processing method according to claim 5, characterized in that: The updating and debugging of the initial mapping table based on the response data includes: Parsing the response data to determine whether there is an error in the response data; If there is no error in the response data, updating the initial mapping table based on the response data; If there is an error in the response data, determining the error type based on the response data; The second mapping relationship corresponding to the response data in the initial mapping table is debugged based on the error type until there is no error in the response data.

7. The data processing method according to claim 1, characterized in that: Before determining the target receiving end interface and the target unique tag in the target request header data based on the first mapping table, the method further includes: In the first mapping table, a unique label is determined through the request header data and the receiving end interface; In the first mapping table, the unique tag is associated with the first mapping relationship.

8. A data processing device, characterized in that: The data processing device comprises: A receiving module, used for receiving target data to be pushed from a target push end, wherein the target data to be pushed includes target request header data and target request body data; A first processing module, configured to determine a target receiving end interface and a target unique tag in the target request header data based on a first mapping table, wherein the first mapping table includes a first mapping relationship between the request header data and the receiving end interface, each first mapping relationship corresponding to a unique tag; A second processing module, configured to determine, based on the target unique tag, in a second mapping table, target received data corresponding to the target request body data, wherein the second mapping table includes second mapping relationships between the request body data and the received data, and each second mapping relationship corresponds to one of the unique tags; The push module is used to push the target receiving data to the target receiving end based on the target receiving end interface.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the data processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the data processing method according to any one of claims 1 to 7 are implemented.

Images