Dynamic data mapping method and system based on field matching

Abstract

The invention discloses a dynamic data mapping method and system based on field matching, and relates to the technical field of data processing. The dynamic data mapping method based on field matching comprises the following steps: maintaining a full data address by a communication management module; dynamically generating a field mapping table by judging whether the lower-level service input can be covered by the output field combination of the upper-level service group; and extracting the required field from the superior service according to the mapping table by using a preset algorithm, and filling the required field into the subordinate input Message. According to the method, the limitation that communication between services needs strict data type matching is solved, flexible connection and efficient data multiplexing are achieved, and the method is suitable for constructing a complex and changeable distributed task process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a dynamic data mapping method and system based on field matching. BACKGROUND

[0002] In a complex distributed task flow, the data dependency relationship between services often changes dynamically and has a complex structure. Traditional methods require strict matching of data types between services, or need to predefine complex data conversion interfaces, which lack flexibility. In addition, when the service topology structure changes or the data format is adjusted, a large amount of redevelopment and testing work is required, which cannot adapt to rapidly changing business requirements. SUMMARY

[0003] The purpose of the present application is to provide a dynamic data mapping method and system based on field matching to at least solve one of the problems existing in the prior art.

[0004] Technical solution: A dynamic data mapping method based on field matching, comprising: In the task configuration stage, the communication management module maintains the full data address of each frame of data of all service nodes in the task flow; In response to the input data requirement of the lower-level service, it is judged whether the input data type is completely the same as the output data type of the directly upper-level service; If not, it is analyzed whether the input data of the lower-level service is composed of part of the fields of the output data of one or more services in the upper-level service group; wherein the upper-level service group includes all direct or indirect upper-level services; When the output data field set of the upper-level service group can cover the input data requirement of the lower-level service, a field mapping table is dynamically generated in the task configuration stage, which is used to record the mapping relationship between each field in the input Message of the current lower-level service and the existing Message field in the upper-level service group; Based on a preset algorithm and in combination with the field mapping table, corresponding field data is requested and extracted from the upper-level service group; The extracted field data is dynamically filled into the input Message of the lower-level service to complete the input data acquisition.

[0005] As a preferred, in response to the input data requirement of the lower-level service, after judging whether the input data type is completely the same as the output data type of the directly upper-level service, it further comprises: If the same, the lower-level service directly acquires input data through the corresponding address in the full data address.

[0006] As preferred, the field mapping table is a structured document, which at least includes: source service identification, source field name, target field name, field type conversion rule, and data extraction path information.

[0007] As preferred, the field mapping table supports dynamic update, and can dynamically adjust the mapping relationship according to the change of service node or data format during the task running, without interrupting the task flow.

[0008] As preferred, analyzing the output data field set of the superior service group can cover the input data requirement of the subordinate service, including the following steps: Traversing all available output fields of the superior service group to form a global available field pool; Matching all required fields of the subordinate service input Message with the global available field pool; If all required fields exist in the global available field pool, it is determined that it can be covered, and the optimal field mapping path is generated.

[0009] As preferred, when generating the optimal field mapping path, the superior service with the nearest data address and the smallest network overhead is preferentially selected as the field source, and the corresponding weight priority information is generated and stored in the field mapping table.

[0010] As preferred, the preset algorithm includes: Based on the protobuf reflection principle, a general serialization and deserialization method is realized.

[0011] As preferred, the full data address includes one or more of: memory physical address, shared memory address, distributed storage system address, or network uniform resource locator address.

[0012] In order to achieve the above purpose, according to another aspect of the present application, a dynamic data mapping device based on field matching is provided.

[0013] The dynamic data mapping device based on field matching according to the present application includes: The data address management module is used for maintaining the full data address of each frame of data of all service nodes in the task flow by the communication management module in the task configuration stage; The judgment module is used for judging whether the input data type of the subordinate service is completely the same as the output data type of the directly superior service in response to the input data requirement of the subordinate service; The field mapping analysis module is used for analyzing whether the input data of the subordinate service is composed of part of the fields of the output data of one or more services in the superior service group if they are not the same; wherein the superior service group includes all direct or indirect superior services; The generating field mapping table module is configured to dynamically generate a field mapping table in a task configuration stage when the output data field set of the upper service group can cover the input data requirement of the lower service, and the field mapping table is used to record the mapping relationship between each field in the input Message of the current lower service and the existing Message field in the upper service group. The extracting field data module is configured to request and extract corresponding field data from the upper service group based on a preset algorithm and in combination with the field mapping table. The data filling module is configured to dynamically fill the extracted field data into the input Message of the lower service to complete the input data acquisition.

[0014] To achieve the above-mentioned purpose, according to another aspect of the present application, an electronic device is provided, which comprises at least one processor and a memory connected with the at least one processor in communication; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the field matching-based dynamic data mapping method according to any one of the present application.

[0015] Beneficial effects: in the embodiment of the present application, the full-amount data address is maintained by the communication management module in a dynamic data mapping manner; the field mapping table is dynamically generated by judging whether the input of the lower service can be covered by the output field combination of the upper service group; the required field is extracted from the upper service according to the mapping table by using the preset algorithm and filled into the lower input Message, so that each service node obtains a field mapping table, thereby realizing the technical effects of flexible connection and efficient data multiplexing, and further solving the technical problems that the traditional method requires strict matching of data types between services, or needs to predefine a complex data conversion interface, lacks flexibility, and when the service topology structure changes or the data format is adjusted, a large amount of re-development and test work is needed, and the method cannot adapt to the rapidly changing business requirements. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a field matching-based dynamic data mapping method flowchart according to an embodiment of the present application; Figure 2 is a structural schematic diagram of a field matching-based dynamic data mapping device according to an embodiment of the present application; and Figure 3 is a structural schematic diagram of an electronic device for a field matching-based dynamic data mapping method according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to enable personnel in the technical field to better understand the scheme of the present application, the technical scheme in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor should fall within the scope of protection of the present application.

[0018] It should be noted that the terms "first", "second" and the like in the description and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0019] In addition, the terms "mount", "set", "provided with", "connected", "connected", "sleeved" should be broadly understood. For example, it can be fixedly connected, detachably connected, or integrally configured; it can be mechanically connected, or electrically connected; it can be directly connected, or indirectly connected through an intermediate medium, or internal communication between two devices, elements or components. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0020] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0021] As shown in Figure 1 According to an embodiment of the present application, a dynamic data mapping method based on field matching is provided, which comprises the following steps S101 to S106: Step S101, task configuration stage, the communication management module maintains the full data address of each frame data of all service nodes in the task flow; The effect of obtaining accurate data can be achieved, thereby providing data basis and guarantee for subsequent work.

[0022] According to an embodiment of the present application, preferably, the full data address comprises one or more of the following: memory physical address, shared memory address, distributed storage system address, or network uniform resource locator address.

[0023] Step S102, in response to the input data requirement of the lower-level service, judging whether the input data type is completely same as the output data type of the directly upper-level service; The accurate data judgment effect can be realized, thereby facilitating subsequent processing.

[0024] Step S103, if not, analyzing whether the input data of the lower-level service is composed of partial fields of the output data of one or more services in the upper-level service group; wherein the upper-level service group comprises all directly or indirectly upper-level services; The accurate data analysis effect can be realized.

[0025] According to the embodiment of the present application, preferably, the output data field set of the upper-level service group can cover the input data requirement of the lower-level service, comprising the following steps: Traversing all available output fields of the upper-level service group to form a global available field pool; Matching all required fields of the lower-level service input Message with the global available field pool; If all required fields exist in the global available field pool, it is determined that it can be covered, and the optimal field mapping path is generated.

[0026] When generating the optimal field mapping path, the upper-level service with the nearest data address and the smallest network overhead is preferentially selected as the field source, and the corresponding weight priority information is generated and stored in the field mapping table.

[0027] According to the embodiment of the present application, preferably, the preset algorithm comprises: A general serialization and deserialization method is realized based on the protobuf reflection principle; Specifically, the field descriptor of any ProtobufMessage is dynamically parsed through the Descriptor and Reflection interfaces of protobuf, and the reading, serialization extraction, deserialization and filling operations of the specified field can be realized without pre-compiled code.

[0028] Step S104, when the output data field set of the upper-level service group can cover the input data requirement of the lower-level service, a field mapping table is dynamically generated in the task configuration stage, and the field mapping table is used to record the mapping relationship between each field in the input Message of the current lower-level service and the existing Message field in the upper-level service group.

[0029] According to the embodiment of the present application, preferably, the field mapping table is a structured document, which at least includes: source service identifier, source field name, target field name, field type conversion rule, and data extraction path information.

[0030] According to the embodiment of the present application, preferably, the field mapping table supports dynamic update, and can dynamically adjust the mapping relationship according to the change of service node or data format during the task running, without interrupting the task flow.

[0031] Step S105, based on a preset algorithm and in combination with the field mapping table, requesting and extracting corresponding field data from the upper service group; Specifically, during runtime, the data assembly module guides the reflection processing engine to perform the following operations: a. Analyzing the field mapping table to determine the data to be obtained from Service_A and Service_B.

[0032] b. According to the data_address provided in the table, accessing the output data of Service_A and Service_B. These data are serialized Protobuf binary streams or Message objects in memory.

[0033] c. For the data of Service_A, the values of id and name fields are dynamically extracted (without pre-compiled code) using the Descriptor and Reflection interfaces of Protobuf.

[0034] d. Similarly, the value of the location field is dynamically extracted from the data of Service_B.

[0035] e. According to the indication of the mapping table, simple type conversion is performed (such as converting the id of int64 to the user_id of int32).

[0036] f. The converted values are also dynamically filled into the corresponding fields (user_id, user_name, user_address) of the blank input Message of Service_C using the reflection interface.

[0037] Step S106, dynamically filling the extracted field data into the input Message of the lower service to complete the input data acquisition.

[0038] The assembled input Message of Service_C is delivered to Service_C to drive it to execute the subsequent business logic.

[0039] According to the embodiment of the present application, preferably, in response to the input data requirement of the lower-level service, after judging whether the input data type is completely identical with the output data type of the directly upper-level service, the method further comprises: If yes, the lower-level service directly acquires the input data through the corresponding address in the full data address.

[0040] The present application solves the field matching problem. After using the above method, even if all data types and all service data output data types of any upper-level service cannot be completely corresponding, as long as the field in the output data type of the upper-level service can cover the input data of the current service, the mapping relationship can be established.

[0041] From the above description, it can be seen that the present application achieves the following technical effects: In the embodiment of the present application, the full data address of each frame of data of all service nodes in the task flow is maintained by the communication management module in the dynamic data mapping manner; the field mapping table is dynamically generated by judging whether the input of the lower-level service can be covered by the output field combination of the upper-level service group; and the required field is extracted from the upper-level service and filled into the lower-level input Message based on the preset algorithm and according to the mapping table, so as to achieve the purpose that each service node obtains a field mapping table, thereby realizing the technical effects of flexible connection and efficient data multiplexing, and further solving the technical problems that the traditional method requires strict matching of data types between services, or needs to predefine a complex data conversion interface, lacks flexibility, and when the service topology structure changes or the data format is adjusted, a large amount of re-development and testing work is needed, and the method cannot adapt to the rapidly changing business requirements.

[0042] As shown in Figure 2 In order to achieve the above purpose, according to another aspect of the present application, a dynamic data mapping device based on field matching is provided. The dynamic data mapping device based on field matching comprises: The data address management module 201 is used for maintaining the full data address of each frame of data of all service nodes in the task flow by the communication management module in the task configuration stage; The judgment module 202 is used for judging whether the input data type is completely identical with the output data type of the directly upper-level service in response to the input data requirement of the lower-level service; The field mapping analysis module 203 is used for analyzing whether the input data of the lower-level service is composed of the partial fields of the output data of one or more services in the upper-level service group if the input data type is not completely identical with the output data type of the directly upper-level service; wherein the upper-level service group comprises all directly or indirectly upper-level services; The generating field mapping table module 204 is configured to dynamically generate a field mapping table in a task configuration stage when the output data field set of the upper service group can cover the input data requirement of the lower service, and the field mapping table is used to record the mapping relationship between each field in the input Message of the current lower service and the existing Message field in the upper service group. The extracting field data module 205 is configured to request and extract corresponding field data from the upper service group based on a preset algorithm and in combination with the field mapping table. The data filling module 206 is configured to dynamically fill the extracted field data into the input Message of the lower service to complete the input data acquisition.

[0043] From the above description, it can be seen that the present application achieves the following technical effects: In the embodiment of the present application, the full-amount data address is maintained by the communication management module in a dynamic data mapping manner; the field mapping table is dynamically generated by judging whether the input of the lower service can be covered by the output field combination of the upper service group; the required field is extracted from the upper service and filled into the lower input Message according to the mapping table by using the general serialization method based on Protobuf reflection, so that each service node obtains a field mapping table, thereby achieving the technical effects of flexible connection and efficient data multiplexing, and further solving the technical problems that the traditional method requires strict matching of data types between services, or needs to predefine a complex data conversion interface, lacks flexibility, and when the service topology structure changes or the data format is adjusted, a large amount of re-development and test work is required, and the method cannot adapt to the rapidly changing business requirements.

[0044] As shown in Figure 3 The electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which are communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0045] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0046] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the field-matching-based dynamic data mapping method.

[0047] In some embodiments, the field-matching-based dynamic data mapping method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the field-matching-based dynamic data mapping method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the field-matching-based dynamic data mapping method by any other appropriate means, such as by means of firmware.

[0048] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0049] Computer programs for implementing the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program, when executed, enables the functions / acts specified in the flowcharts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package and partially on a remote machine or entirely on a remote machine or server.

[0050] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0051] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0052] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0053] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0054] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, executed in sequence, or executed in different orders, as long as the desired results of the technical solutions of the present disclosure can be achieved, and the present disclosure is not limited herein.

[0055] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A dynamic data mapping method based on field matching, characterized in that, include: During the task configuration phase, the communication management module maintains the full data address of each frame of data for all service nodes in the task process. In response to the input data request of the lower-level service, determine whether its input data type is exactly the same as the output data type of the direct upper-level service; If they are not the same, then analyze whether the input data of the lower-level service is composed of a combination of partial fields of the output data of one or more services in the upper-level service group; wherein, the upper-level service group includes all direct or indirect upper-level services; When the set of output data fields of the upper-level service group can cover the input data requirements of the lower-level service, a field mapping table is dynamically generated during the task configuration phase. The field mapping table is used to record the mapping relationship between each field in the input Message of the current lower-level service and the existing Message fields in the upper-level service group. Based on a preset algorithm and in conjunction with the field mapping table, the corresponding field data is requested and extracted from the superior service group; The extracted field data is dynamically populated into the input Message of the lower-level service to complete the input data acquisition.

2. The dynamic data mapping method based on field matching according to claim 1, characterized in that, In response to the input data requests of lower-level services, after determining whether the input data type is exactly the same as the output data type of the direct superior service, the process also includes: If they are the same, the lower-level service directly obtains the input data through the corresponding address in the full data address.

3. The dynamic data mapping method based on field matching according to claim 1, characterized in that, The field mapping table is a structured document that includes at least: source service identifier, source field name, target field name, field type conversion rules, and data extraction path information.

4. The dynamic data mapping method based on field matching according to claim 1, characterized in that, The field mapping table supports dynamic updates, enabling it to dynamically adjust the mapping relationship during task execution based on changes in service nodes or data formats without interrupting the task flow.

5. The dynamic data mapping method based on field matching according to claim 1, characterized in that, Analyzing whether the output data field set of the parent service group can cover the input data requirements of the lower service includes the following steps: Iterate through all available output fields of the parent service group to form a global pool of available fields; Match all required fields of the input Message from the lower-level service with the global available field pool; If all required fields exist in the global available field pool, it is determined that the field can be covered, and the optimal field mapping path is generated.

6. The dynamic data mapping method based on field matching according to claim 5, characterized in that, When generating the optimal field mapping path, the upstream service with the closest data address and the lowest network overhead is selected as the field source, and the corresponding weight and priority information is generated and stored in the field mapping table.

7. The dynamic data mapping method based on field matching according to claim 1, characterized in that, The algorithm based on the preset algorithm includes: A general serialization and deserialization method is implemented based on the protobuf reflection principle.

8. The dynamic data mapping method based on field matching according to claim 1, characterized in that, The full data address includes one or more of the following: physical memory address, shared memory address, distributed storage system address, or network Uniform Resource Locator address.

9. A dynamic data mapping device based on field matching, characterized in that, include: The data address management module is used during the task configuration phase. The communication management module maintains the full data address of each frame of data for all service nodes in the task process. The judgment module is used to respond to the input data requirements of the lower-level service and determine whether its input data type is exactly the same as the output data type of the direct upper-level service; The field mapping analysis module is used to analyze whether the input data of the lower-level service is composed of a combination of partial fields of the output data of one or more services in the upper-level service group if they are different; wherein, the upper-level service group includes all direct or indirect upper-level services. The field mapping table generation module is used to dynamically generate a field mapping table during the task configuration stage when the output data field set of the upper-level service group can cover the input data requirements of the lower-level service. The field mapping table is used to record the mapping relationship between each field in the input Message of the current lower-level service and the existing Message fields in the upper-level service group. The field data extraction module is used to request and extract corresponding field data from the superior service group based on a preset algorithm and in conjunction with the field mapping table. The data filling module is used to dynamically fill the extracted field data into the input Message of the lower-level service to complete the input data acquisition.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the field-matching-based dynamic data mapping method according to any one of claims 1-8.

Images