A Dynamic Data Mapping Method and System Based on Field Matching

Abstract

This invention discloses a dynamic data mapping method and system based on field matching, relating to the field of data processing technology. The dynamic data mapping method based on field matching includes: maintaining the full data address by a communication management module; dynamically generating a field mapping table by determining whether the input of a lower-level service can be covered by a combination of output fields from a higher-level service group; and using a preset algorithm to extract the required fields from the higher-level service according to the mapping table and fill them into the lower-level input Message. This invention solves the limitation of strict data type matching required for inter-service communication, achieving flexible connection and efficient data reuse, and is suitable for building complex and ever-changing distributed task processes.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a dynamic data mapping method and system based on field matching. Background Technology

[0002] In complex distributed task flows, the data dependencies between services are often dynamically changing and structurally complex. Traditional methods require strict matching of data types between services or pre-defined complex data conversion interfaces, lacking flexibility; furthermore, when the service topology changes or the data format is adjusted, a large amount of redevelopment and testing work is required, making it unable to adapt to rapidly changing business needs. Summary of the Invention

[0003] Purpose of the invention: To provide a dynamic data mapping method and system based on field matching, so as to at least solve one of the problems existing in the prior art.

[0004] Technical solution: A dynamic data mapping method based on field matching, comprising: During the task configuration phase, the communication management module maintains the full data address of each frame of data for all service nodes in the task process. In response to the input data request of the lower-level service, determine whether its input data type is exactly the same as the output data type of the direct upper-level service; If they are not the same, then analyze whether the input data of the lower-level service is composed of a combination of partial fields of the output data of one or more services in the upper-level service group; wherein, the upper-level service group includes all direct or indirect upper-level services; When the set of output data fields of the upper-level service group can cover the input data requirements of the lower-level service, a field mapping table is dynamically generated during the task configuration phase. The field mapping table is used to record the mapping relationship between each field in the input Message of the current lower-level service and the existing Message fields in the upper-level service group. Based on a preset algorithm and in conjunction with the field mapping table, the corresponding field data is requested and extracted from the superior service group; The extracted field data is dynamically populated into the input Message of the lower-level service to complete the input data acquisition.

[0005] As a preferred option, in response to the input data requirements of lower-level services, after determining whether the input data type is exactly the same as the output data type of the direct superior service, the following steps are also included: If they are the same, the lower-level service directly obtains the input data through the corresponding address in the full data address.

[0006] Preferably, the field mapping table is a structured document that includes at least: source service identifier, source field name, target field name, field type conversion rules, and data extraction path information.

[0007] Preferably, the field mapping table supports dynamic updates, enabling the mapping relationship to be dynamically adjusted during task execution based on changes in service nodes or data formats, without interrupting the task flow.

[0008] Preferably, analyzing the set of output data fields of the upper-level service group can cover the input data requirements of the lower-level services, including the following steps: Iterate through all available output fields of the parent service group to form a global pool of available fields; Match all required fields of the input Message from the lower-level service with the global available field pool; If all required fields exist in the global available field pool, it is determined that the field can be covered, and the optimal field mapping path is generated.

[0009] Preferably, when generating the optimal field mapping path, the upstream service with the closest data address and the lowest network overhead is selected as the field source, and the corresponding weight and priority information is generated and stored in the field mapping table.

[0010] Preferably, the algorithm based on a preset algorithm includes: A general serialization and deserialization method is implemented based on the protobuf reflection principle.

[0011] Preferably, the full data address includes one or more of the following: physical memory address, shared memory address, distributed storage system address, or network Uniform Resource Locator address.

[0012] To achieve the above objectives, according to another aspect of this application, a dynamic data mapping device based on field matching is provided.

[0013] The field-matching-based dynamic data mapping apparatus according to this application includes: The data address management module is used during the task configuration phase. The communication management module maintains the full data address of each frame of data for all service nodes in the task process. The judgment module is used to respond to the input data requirements of the lower-level service and determine whether its input data type is exactly the same as the output data type of the direct upper-level service; The field mapping analysis module is used to analyze whether the input data of the lower-level service is composed of a combination of partial fields of the output data of one or more services in the upper-level service group if they are different; wherein, the upper-level service group includes all direct or indirect upper-level services. The field mapping table generation module is used to dynamically generate a field mapping table during the task configuration stage when the output data field set of the upper-level service group can cover the input data requirements of the lower-level service. The field mapping table is used to record the mapping relationship between each field in the input Message of the current lower-level service and the existing Message fields in the upper-level service group. The field data extraction module is used to request and extract corresponding field data from the superior service group based on a preset algorithm and in conjunction with the field mapping table. The data filling module is used to dynamically fill the extracted field data into the input Message of the lower-level service to complete the input data acquisition.

[0014] To achieve the above objectives, according to another aspect of this application, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the field-matching-based dynamic data mapping method according to any one of the present invention.

[0015] Beneficial effects: In this embodiment, a dynamic data mapping method is adopted, with the communication management module maintaining the full data address. By determining whether the input of the lower-level service can be covered by the output field combination of the upper-level service group, a field mapping table is dynamically generated. Using a preset algorithm, the required fields are extracted from the upper-level service according to the mapping table and filled into the lower-level input Message, achieving the goal of obtaining a field mapping table for each service node. This realizes the technical effect of flexible connection and efficient data reuse, thereby solving the technical problems of traditional methods that require strict matching of data types between services or require pre-defined complex data conversion interfaces, lacking flexibility; and the need for a large amount of redevelopment and testing work when the service topology changes or the data format is adjusted, making it unable to adapt to rapidly changing business needs. Attached Figure Description

[0016] Figure 1 This is a flowchart of a dynamic data mapping method based on field matching according to an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a dynamic data mapping device based on field matching according to an embodiment of this application; and Figure 3 This is a schematic diagram of the structure of an electronic device according to the field matching-based dynamic data mapping method of the present application. Detailed Implementation

[0017] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0018] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0019] Furthermore, the terms "installation," "setup," "equipped with," "connection," "linking," and "socketing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral structure; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium, or an internal connection between two devices, components, or parts. Those skilled in the art can understand the specific meaning of these terms in this invention according to the specific circumstances.

[0020] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] like Figure 1 As shown in the embodiment of the present invention, a dynamic data mapping method based on field matching is provided, the method comprising the following steps S101 to S106: Step S101, Task Configuration Phase: The communication management module maintains the full data address of each frame of data for all service nodes in the task flow. It can achieve the effect of obtaining accurate data, thereby providing a data foundation and guarantee for subsequent work.

[0022] According to an embodiment of the present invention, preferably, the full data address includes one or more of the following: physical memory address, shared memory address, distributed storage system address, or network uniform resource locator address.

[0023] Step S102: In response to the input data request of the lower-level service, determine whether its input data type is exactly the same as the output data type of the direct upper-level service; It can achieve accurate data judgment, thus facilitating subsequent processing.

[0024] Step S103: If they are not the same, analyze whether the input data of the lower-level service is composed of a combination of some fields of the output data of one or more services in the upper-level service group; wherein, the upper-level service group includes all direct or indirect upper-level services. It can achieve accurate data analysis results.

[0025] According to an embodiment of the present invention, preferably, analyzing the output data field set of the upper-level service group to cover the input data requirements of the lower-level service includes the following steps: Iterate through all available output fields of the parent service group to form a global pool of available fields; Match all required fields of the input Message from the lower-level service with the global available field pool; If all required fields exist in the global available field pool, it is determined that the field can be covered, and the optimal field mapping path is generated.

[0026] When generating the optimal field mapping path, the upstream service with the closest data address and the lowest network overhead is selected as the field source, and the corresponding weight and priority information is generated and stored in the field mapping table.

[0027] According to an embodiment of the present invention, preferably, the step of basing the algorithm on a preset algorithm includes: A general serialization and deserialization method based on the protobuf reflection principle; Specifically, by using protobuf's Descriptor and Reflection interfaces, the field descriptors of any ProtobufMessage can be dynamically parsed, enabling the reading, serialization extraction, deserialization, and padding of specified fields without the need for pre-compiled code.

[0028] Step S104: When the set of output data fields of the upper-level service group can cover the input data requirements of the lower-level service, a field mapping table is dynamically generated during the task configuration stage. The field mapping table is used to record the mapping relationship between each field in the input Message of the current lower-level service and the existing Message fields in the upper-level service group.

[0029] According to an embodiment of the present invention, preferably, the field mapping table is a structured document, which includes at least: source service identifier, source field name, target field name, field type conversion rules, and data extraction path information.

[0030] According to an embodiment of the present invention, preferably, the field mapping table supports dynamic updates, and can dynamically adjust the mapping relationship according to changes in service nodes or data formats during task execution without interrupting the task flow.

[0031] Step S105: Based on the preset algorithm and in conjunction with the field mapping table, request and extract the corresponding field data from the superior service group; Specifically, at runtime, the data assembly module guides the reflection processing engine to perform the following operations: a. Parse the field mapping table to determine which services, Service_A and Service_B, need to be used to obtain data.

[0032] b. Access the output data of Service_A and Service_B based on the data_address provided in the table. This data is either a serialized Protobuf binary stream or a Message object in memory.

[0033] c. For the data of Service_A, use the Protobuf Descriptor and Reflection interfaces to dynamically (without pre-compiling code) extract the values ​​of the id and name fields.

[0034] d. Similarly, dynamically extract the value of the location field from the data of Service_B.

[0035] e. Perform simple type conversions as indicated by the mapping table (e.g., convert an int64 id to an int32 user_id).

[0036] f. The converted values ​​are then dynamically populated into the corresponding fields (user_id, user_name, user_address) of the blank input Message in Service_C using the reflection interface.

[0037] Step S106: Dynamically fill the extracted field data into the input Message of the lower-level service to complete the input data acquisition.

[0038] The assembled Service_C input Message is delivered to Service_C to drive it to execute subsequent business logic.

[0039] According to an embodiment of the present invention, preferably, after determining whether the input data type of a lower-level service is completely identical to the output data type of its direct superior service in response to the input data request of the lower-level service, the method further includes: If they are the same, the lower-level service directly obtains the input data through the corresponding address in the full data address.

[0040] This application solves the field matching problem. After adopting the above method, even if all data types and all service data output data types of any superior cannot completely correspond, as long as the field in the output data type of all nodes output by the superior can cover the input data of the current service, a mapping relationship can be established.

[0041] As can be seen from the above description, this application achieves the following technical effects: In this embodiment, a dynamic data mapping approach is adopted, with the communication management module maintaining all data addresses. A field mapping table is dynamically generated by determining whether the input of a lower-level service can be covered by the output field combination of the upper-level service group. Using a preset algorithm, the required fields are extracted from the upper-level service based on the mapping table and filled into the lower-level input Message. This achieves the goal of obtaining a field mapping table for each service node, thus realizing the technical effects of flexible connection and efficient data reuse. This solves the technical problems of traditional methods requiring strict matching of data types between services or pre-defined complex data conversion interfaces, lacking flexibility; and requiring extensive redevelopment and testing work when the service topology changes or the data format is adjusted, making it unable to adapt to rapidly changing business needs.

[0042] like Figure 2 As shown, to achieve the above objectives, according to another aspect of this application, a dynamic data mapping apparatus based on field matching is provided. This dynamic data mapping apparatus based on field matching includes: The data address management module 201 is used in the task configuration phase. The communication management module maintains the full data address of each frame of data of all service nodes in the task process. The judgment module 202 is used to respond to the input data requirements of the lower-level service and determine whether its input data type is exactly the same as the output data type of the direct upper-level service; The field mapping analysis module 203 is used to analyze whether the input data of the lower-level service is composed of a combination of partial fields of the output data of one or more services in the upper-level service group if they are not the same; wherein, the upper-level service group includes all direct or indirect upper-level services. The field mapping table generation module 204 is used to dynamically generate a field mapping table during the task configuration stage when the output data field set of the upper-level service group can cover the input data requirements of the lower-level service. The field mapping table is used to record the mapping relationship between each field in the input Message of the current lower-level service and the existing Message fields in the upper-level service group. The field data extraction module 205 is used to request and extract corresponding field data from the superior service group based on a preset algorithm and in conjunction with the field mapping table. The data filling module 206 is used to dynamically fill the extracted field data into the input Message of the lower-level service to complete the input data acquisition.

[0043] As can be seen from the above description, this application achieves the following technical effects: In this embodiment, a dynamic data mapping approach is adopted, with the communication management module maintaining the full data addresses. A field mapping table is dynamically generated by determining whether the input of a lower-level service can be covered by the output field combination of the upper-level service group. Using a general serialization method based on Protobuf reflection, the required fields are extracted from the upper-level service according to the mapping table and filled into the lower-level input Message. This achieves the goal of obtaining a field mapping table for each service node, thus realizing the technical effects of flexible connection and efficient data reuse. This solves the technical problems of traditional methods requiring strict matching of data types between services or pre-defined complex data conversion interfaces, lacking flexibility; and requiring extensive redevelopment and testing work when the service topology changes or the data format is adjusted, making it unable to adapt to rapidly changing business needs.

[0044] like Figure 3 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0045] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0046] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as dynamic data mapping methods based on field matching.

[0047] In some embodiments, the field-matching-based dynamic data mapping method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the field-matching-based dynamic data mapping method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the field-matching-based dynamic data mapping method by any other suitable means (e.g., by means of firmware).

[0048] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0049] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0050] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0051] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0052] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LA), wide area networks (WA), blockchain networks, and the Internet.

[0053] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0054] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0055] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A dynamic data mapping method based on field matching, characterized in that, include: During the task configuration phase, the communication management module maintains the full data address of each frame of data for all service nodes in the task process. In response to the input data request of the lower-level service, determine whether its input data type is exactly the same as the output data type of the direct upper-level service; If they are not the same, then analyze whether the input data of the lower-level service is composed of a combination of partial fields of the output data of one or more services in the upper-level service group; wherein, the upper-level service group includes all direct or indirect upper-level services; When the set of output data fields of the upper-level service group can cover the input data requirements of the lower-level service, a field mapping table is dynamically generated during the task configuration phase. The field mapping table is used to record the mapping relationship between each field in the input Message of the current lower-level service and the existing Message fields in the upper-level service group. Based on a preset algorithm and in conjunction with the field mapping table, the corresponding field data is requested and extracted from the superior service group; The algorithm based on the preset algorithm includes: A general serialization and deserialization method based on the protobuf reflection principle; By dynamically parsing the field descriptors of any Protobuf Message through the Descriptor and Reflection interfaces of protobuf, it is possible to read, serialize, extract, deserialize and fill specified fields without the need for pre-compiled code; The extracted field data is dynamically populated into the input Message of the lower-level service to complete the input data acquisition.

2. The dynamic data mapping method based on field matching according to claim 1, characterized in that, In response to the input data requests of lower-level services, after determining whether the input data type is exactly the same as the output data type of the direct superior service, the process also includes: If they are the same, the lower-level service directly obtains the input data through the corresponding address in the full data address.

3. The dynamic data mapping method based on field matching according to claim 1, characterized in that, The field mapping table is a structured document that includes at least: source service identifier, source field name, target field name, field type conversion rules, and data extraction path information.

4. The dynamic data mapping method based on field matching according to claim 1, characterized in that, The field mapping table supports dynamic updates, enabling it to dynamically adjust the mapping relationship during task execution based on changes in service nodes or data formats without interrupting the task flow.

5. The dynamic data mapping method based on field matching according to claim 1, characterized in that, Analyzing whether the output data field set of the parent service group can cover the input data requirements of the lower service includes the following steps: Iterate through all available output fields of the parent service group to form a global pool of available fields; Match all required fields of the input Message from the lower-level service with the global available field pool; If all required fields exist in the global available field pool, it is determined that the field can be covered, and the optimal field mapping path is generated.

6. The dynamic data mapping method based on field matching according to claim 5, characterized in that, When generating the optimal field mapping path, the upstream service with the closest data address and the lowest network overhead is selected as the field source, and the corresponding weight and priority information is generated and stored in the field mapping table.

7. The dynamic data mapping method based on field matching according to claim 1, characterized in that, The full data address includes one or more of the following: physical memory address, shared memory address, distributed storage system address, or network Uniform Resource Locator address.

8. A dynamic data mapping device based on field matching, characterized in that, include: The data address management module is used during the task configuration phase. The communication management module maintains the full data address of each frame of data for all service nodes in the task process. The judgment module is used to respond to the input data requirements of the lower-level service and determine whether its input data type is exactly the same as the output data type of the direct upper-level service; The field mapping analysis module is used to analyze whether the input data of the lower-level service is composed of a combination of partial fields of the output data of one or more services in the upper-level service group if they are different; wherein, the upper-level service group includes all direct or indirect upper-level services. The field mapping table generation module is used to dynamically generate a field mapping table during the task configuration stage when the output data field set of the upper-level service group can cover the input data requirements of the lower-level service. The field mapping table is used to record the mapping relationship between each field in the input Message of the current lower-level service and the existing Message fields in the upper-level service group. The field data extraction module is used to request and extract corresponding field data from the superior service group based on a preset algorithm and in conjunction with the field mapping table. The algorithm based on the preset algorithm includes: A general serialization and deserialization method based on the protobuf reflection principle; By dynamically parsing the field descriptors of any Protobuf Message through the Descriptor and Reflection interfaces of protobuf, it is possible to read, serialize, extract, deserialize and fill specified fields without the need for pre-compiled code; The data filling module is used to dynamically fill the extracted field data into the input Message of the lower-level service to complete the input data acquisition.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the field-matching-based dynamic data mapping method according to any one of claims 1-7.

Images