Data format conversion method, device, equipment and medium based on dynamic lazy loading

Abstract

The present invention discloses a data format conversion method, device, equipment and medium based on dynamic lazy loading. The method comprises: in response to a data transmission request from a client, obtaining original data and target format information; parsing the original data to generate a data object; if the original data needs to be format converted, searching a plug-in library for a format conversion plug-in that matches the target format information as a target conversion plug-in, and loading the target conversion plug-in into memory; after the target conversion plug-in is loaded, calling a plug-in method through reflection so that the target conversion plug-in obtains the data object, maps and serializes the data object, and obtains the target data to be transmitted. The present invention effectively avoids the loading of a large number of potentially idle format conversion plug-ins during system initialization, significantly reduces system resource usage, and significantly improves the operating efficiency and startup speed of the system.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data format conversion method, device, equipment and medium based on dynamic lazy loading. Background Art

[0002] In today's complex data communication and processing environment, data exchange between different systems inevitably faces the challenge of converting data formats to meet the specific requirements of the recipient. Traditional data format conversion methods often load all possible format conversion modules during system initialization. This approach has significant drawbacks: First, the loading of a large number of potentially unused modules significantly consumes system resources, resulting in slow system startup and significantly reduced operational efficiency. Second, whenever new data format conversion logic is needed, the entire system must be redeployed and modified, resulting in limited flexibility and difficulty keeping up with the rapidly changing business pace. Especially when sending data to a Kafka cluster, balancing efficiency and flexibility, properly handling data in various formats, and meeting the diverse needs of different customers has become a pressing challenge in this field. Summary of the Invention

[0003] In order to solve the above problems existing in the prior art, the present invention provides a data format conversion method, device, equipment and medium based on dynamic lazy loading. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0004] A first aspect of the present invention provides a data format conversion method based on dynamic lazy loading, which is applied to a main engine. The method includes:

[0005] Responding to a data transmission request from a client, obtaining original data and target format information;

[0006] Parsing the original data to generate a data object;

[0007] If the format of the original data needs to be converted, a format conversion plug-in matching the target format information is searched in the plug-in library as a target conversion plug-in, and the target conversion plug-in is loaded into the memory;

[0008] After the target conversion plug-in is loaded, the plug-in method is called through reflection to enable the target conversion plug-in to obtain the data object, map and serialize the data object, and obtain the target data to be transmitted.

[0009] The method provided by the present invention uses a dynamic lazy loading mechanism to load the corresponding conversion plug-in only when data format conversion is actually needed, effectively avoiding the loading of a large number of possibly idle format conversion plug-ins during system initialization, greatly reducing system resource usage, and significantly improving the system's operating efficiency and startup speed.

[0010] In a possible implementation, the method further includes:

[0011] Identifying whether the original data is valid;

[0012] If the original data is valid, determining a valid data type;

[0013] It is determined whether format conversion needs to be performed on the original data based on the valid data type and the target format information.

[0014] In a possible implementation, parsing the original data to generate a data object includes:

[0015] Parsing the original data to obtain DML data, DDL data, and SQL data;

[0016] The DML data, the DDL data and the SQL data are used as data, the operation statement framework is encapsulated into an operation data object, the data dictionary is encapsulated into a dictionary data object, and the business data is encapsulated into a business data object.

[0017] In a possible implementation, the method further includes:

[0018] When the main engine is started, all plug-in jar files in the plug-in directory are read and loaded into the Java virtual machine; each plug-in jar file corresponds to a conversion plug-in, and the plug-in jar file includes data format conversion logic, business processing methods and plug-in functions.

[0019] In a possible implementation, the method further includes:

[0020] When a new customer data conversion customization requirement is added, a new plug-in project created by a developer in an integrated development environment is obtained, and the plug-in project is packaged and deployed to the plug-in directory.

[0021] In a possible implementation, the method further includes:

[0022] When the plug-in library is initialized, unique identification information is assigned to each format conversion plug-in, and a mapping table between the unique identification information and target format information is established;

[0023] The step of searching the plug-in library for a format conversion plug-in that matches the target format information as the target conversion plug-in includes:

[0024] Searching the mapping table for the unique identification information that has a mapping relationship with the target format information;

[0025] The format conversion plug-in corresponding to the unique identification information is used as the target conversion plug-in.

[0026] In a possible implementation, the method further includes:

[0027] receiving the target data sent by the target conversion plug-in after verifying the target data, and sending the target data to a target recipient; or

[0028] Receive error report information sent by the target conversion plug-in after the target data fails to be verified, and generate a processing solution based on the error report information.

[0029] A second aspect of the present invention provides a data format conversion device based on dynamic lazy loading, which is applied to a main engine. The device includes:

[0030] An acquisition module, configured to respond to a data transmission request from a client and acquire original data and target format information;

[0031] An object generation module, configured to parse the original data and generate a data object;

[0032] a search and loading module, configured to search for a target conversion plug-in that matches the target format information if format conversion is required for the original data, and load the searched target conversion plug-in into a memory;

[0033] The plug-in calling module is used to call the plug-in method through reflection after the target conversion plug-in is loaded, so that the target conversion plug-in obtains the data object, maps and serializes the data object, and obtains the target data to be transmitted.

[0034] The third aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, a data format conversion method based on dynamic lazy loading provided by the first aspect of the present invention is implemented.

[0035] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for data format conversion based on dynamic lazy loading provided by the first aspect of the present invention is implemented.

[0036] For the specific description of the second to fourth aspects of the present invention and their various implementations, reference can be made to the detailed description of the first aspect and its various implementations; and for the beneficial effects of the second to fourth aspects and their various implementations, reference can be made to the analysis of the beneficial effects of the first aspect and its various implementations, which will not be repeated here.

[0037] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of a data format conversion method based on dynamic lazy loading according to an embodiment of the present invention;

[0039] Figure 2 This is a structural block diagram of a data format conversion device based on dynamic lazy loading according to an embodiment of the present invention;

[0040] Figure 3 This is a block diagram of the internal structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0042] An embodiment of the present invention provides a data format conversion method based on dynamic lazy loading. The method can be executed by an electronic device, which can be a server or a terminal device. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smartphone, a tablet computer, a desktop computer, a wearable device, etc., but is not limited to these. Of course, the electronic device can also be an indoor light source control device or a photocatalytic air purifier.

[0043] Figure 1 This is a flow chart of a data format conversion method based on dynamic lazy loading applied to the main engine provided in this embodiment. Figure 1 As shown, the main process of the method is described as follows (steps S101 to S104):

[0044] Step S101, in response to a data transmission request from a client, obtaining original data and target format information;

[0045] Step S102, parsing the original data to generate a data object;

[0046] Step S103: If the original data needs to be format converted, a format conversion plug-in matching the target format information is searched in the plug-in library as a target conversion plug-in, and the target conversion plug-in is loaded into the memory;

[0047] Step S104: After the target conversion plug-in is loaded, the plug-in method is called through reflection to enable the target conversion plug-in to obtain the data object, map and serialize the data object, and obtain the target data to be transmitted.

[0048] In this embodiment, the main engine can be the Kafka Data Engine (JDDM). As the key producer engine for sending standardized data to the Kafka cluster, the Kafka Data Engine (JDDM) listens on a designated port (socket) and awaits data transmission requests from clients. It possesses powerful parallel processing capabilities and can simultaneously receive standard data blocks sent by clients on different servers across wide and local area networks. These data blocks include DDL (a data dictionary containing detailed table information such as user names, table names, tablespaces, column names, column types, column lengths, and column precision), DML (detailed data related to table operations such as insert, update, and delete), and SQL (primarily referring to DDL SQL changes, such as database operation instructions such as adding or removing columns from a table, modifying column types, and adjusting column lengths and precision).

[0049] When the system starts, the main engine's plug-in management module first initializes the plug-in library and builds an efficient plug-in index to facilitate subsequent rapid search for the required target conversion plug-in. Specifically, during plug-in library initialization, each format conversion plug-in is assigned unique identification information (such as the plug-in name, plug-in ID, etc.), and this unique identification information is associated with the plug-in function (such as supported data formats). A mapping table is then established between the unique identification information and the target format information. This mapping table can be a hash table or index structure, with the target format information as the key and the corresponding unique identification information as the value.

[0050] At that time, when the plug-in management module receives the target format information and needs to find a matching target conversion plug-in, it can search for the unique identification information that has a mapping relationship with the target format information in the pre-established mapping table, and use the format conversion plug-in corresponding to the unique identification information as the target conversion plug-in. The corresponding target conversion plug-in can be quickly retrieved in the hash table directly through the target format information.

[0051] During the plugin loading phase, the main engine listens for plugin files in the specified path. When a new plugin file is detected, it loads it, parses its supported data formats, and dynamically updates the plugin index. This ensures the plugin index is always up-to-date and can quickly respond to new plugin additions or updates.

[0052] During the plug-in development phase, metadata is defined for each format conversion plug-in, including information such as the data formats supported by the plug-in and descriptions of its functionality. When new customer data conversion requirements are added, the new plug-in project / project created by the developer in an integrated development environment (such as Eclipse / IDEA) is obtained, packaged, and deployed to the plug-in directory.

[0053] It's important to note that the public code for the format conversion plugin can be directly pulled from Git. Based on this public code, developers can focus on implementing customer-specific data conversion functionality. Because the public code already provides the foundational functionality, developers can quickly implement customized requirements without having to write all the code from scratch.

[0054] When business requirements change and new data format conversion logic needs to be added, you only need to easily add the new format conversion plug-in to the plug-in library. The format conversion plug-in can be automatically loaded and put into use when needed, without the need for large-scale redeployment and modification of the entire system, greatly enhancing the system's adaptability to business changes.

[0055] In this embodiment, the plugin directory refers to the Module directory, which stores files (such as Jar files) for format conversion plugins. This directory is the core directory used by the main engine to load and manage format conversion plugins. The plugin directory contains multiple Jar files, each of which represents a separate module. Plugin projects are independent of each other, effectively isolating their code and greatly facilitating project and version management. Different plugins are independent and do not interfere with each other during development, maintenance, and upgrades, making it easier to maintain separate version control and management for each plugin, effectively reducing project management complexity.

[0056] This model enables parallel development for business needs across multiple industries. Different development teams or individuals can simultaneously develop corresponding plug-ins for different industry needs without interfering with each other. This helps accelerate overall project development, more quickly meet diverse customer needs, and significantly improves project delivery efficiency. The plug-in model is designed to meet the personalized data conversion needs of different customers and can be developed by different junior R&D personnel. By working in conjunction with JDDM (Kafka's primary data engine), it can effectively meet the complex and ever-changing business needs of customers across different industries.

[0057] The plug-in model effectively lowers the R&D threshold. Ordinary R&D personnel only need to focus on the specific business logic implementation of data conversion without having to delve into the complex architecture and operation mechanism of the entire Kafka data engine. This allows more ordinary R&D personnel to participate in project development, greatly expanding the scope of available R&D resources.

[0058] When the main engine starts, it reads all the plug-in jar files in the plug-in directory and loads them into the Java Virtual Machine (JVM). Each plug-in jar file corresponds to a conversion plug-in, and the plug-in jar file includes data format conversion logic, business processing methods, and plug-in functions.

[0059] It can be seen that in the main engine startup process, the clever method of dynamically loading the jar package can be used to seamlessly integrate the business processing module of the JDDM data engine, thereby minimizing interference with the original architecture of the main engine.

[0060] In this embodiment, the main engine uses a DDL parser, a DML parser, and an SQL change parser to parse the raw data sent by the client, obtaining DDL data, DML data, and SQL data. Using this DML data, DDL data, and SQL data as data, the main engine encapsulates the operation statement framework into operation data objects, the data dictionary into dictionary data objects, and the business data into business data objects.

[0061] Determine the table and field names in the operation statement and the associated data dictionary. The data dictionary is a set of system tables in the database management system (DBMS) that stores database metadata. Convert the data dictionary into a dictionary data object, which contains information such as the converted table structure and attributes.

[0062] An operation statement framework typically refers to a set of predefined logic for generating and executing operation statements (such as SQL queries and API requests). By encapsulating common operations (such as database queries, API calls, and data processing) into reusable components, it simplifies the development process and improves code readability and maintainability. The core goals of the operation statement framework are to reduce duplication, improve development efficiency, and support dynamic generation and execution of operations. Business data is the actual data stored by users in the database, which supports the operation of business logic and applications.

[0063] In this embodiment, the main engine judges the format of the original data based on pre-set format recognition rules, determines whether format conversion is required and the specific target data format, and indicates the direction for data processing.

[0064] Specifically, first identify whether the original data is valid and ensure that the original data complies with predefined rules, such as format, protocol, business logic, etc., to avoid processing errors or malicious data and ensure system stability; if the original data is valid, determine the valid data type, such as structured data (JSON / XML), binary protocol, free text, etc., to provide a decision basis for subsequent format conversion; then determine whether the original data needs to be format converted based on the valid data type and target format information. If the compared data type is incompatible with the target format, the format conversion is triggered.

[0065] If the original data is DDL information for a database table structure, and the target customer requires Avro for data storage, the main engine accurately determines the need for format conversion and passes key information, including the target data format (AVRO), to the format conversion plug-in. Using four layers of logic, including syntax feature recognition, metadata comparison, protocol matching, and intelligent inference, the format conversion process is accurately triggered when the original data is DDL information and the target format is Avro.

[0066] After receiving the data object (Java object) passed by the main engine, the target conversion plug-in first uses lexical analysis and syntax analysis techniques to conduct in-depth analysis of the data. For data containing DDL information, it can accurately parse key content such as table structure and field definitions. For DML data, it can accurately analyze the operation type (insert, modify, delete) and specific data values, breaking the data into basic elements for subsequent processing.

[0067] Based on the target data format (such as AVRO, JSON, ProtocolBuffer, TEXT, Bytes, etc.) and specific customer needs, the plugin meticulously formulates corresponding data mapping rules. For example, it cleverly maps database table column names to JSON key names and precisely converts database data types to AVRO-specified data types. These mapping rules can be defined through flexible configuration files, concise annotations, or built-in code, ensuring accurate data conversion to the target format from the source.

[0068] When converting to JSON format, the data is assembled into key-value pairs in an orderly manner according to the mapping rules to construct a standard string that conforms to the JSON syntax specification; when converting to AVRO format, full use is made of AVRO's serialization tools and the data is serialized strictly according to AVRO's schema definition.

[0069] For digital dictionaries, call the public List of the plug-in project <object>dataDictionaryConversionByUserCustom(TableInfoVo tableInfoVo,List <tabblecolumnvo>nowColList,List <tablecolumnvo>cacheColList){…}; method performs data dictionary-related business processing according to the data dictionary conversion rules.

[0070] For SQL statements, call the public List in the plug-in project <object>The sqlStringConversionByUserCustom(String schemaName,String tableName,StringsqlString){…}; method performs related business processing of SQL statements according to the SQL statement conversion rules.

[0071] For DML data (Insert / Update / Delete), call the public List in the plug-in project <object>The dmlDataConversionByUserCustom(PackageRetrunVo packageRetrunVo){…}; method performs related business processing of DML data according to the DML data conversion rules.

[0072] After obtaining DML data, data dictionary, and SQL data, the format conversion plug-in converts the format according to the customer-defined rules and sends the converted DML data, data dictionary, and SQL data to the list. <object>The data type is returned to the main engine.

[0073] This embodiment splits the data block into an operation statement framework, a data dictionary, and business data, and performs format conversion separately, making the conversion more flexible. Furthermore, the operation statement framework, data dictionary, and business data are converted into data objects, which are passed as parameters and packaged as a whole, facilitating data synchronization.

[0074] Preferably, after converting the original data format, the target conversion plug-in verifies the target data to ensure its integrity and accuracy. For example, it carefully checks whether JSON data conforms to JSON syntax rules and whether AVRO data perfectly matches the AVRO schema. If the verification passes, the target conversion plug-in sends the target data to the main engine's data transmission module, which then sends the target data to the intended recipient. If the verification fails, the target conversion plug-in generates an error report and feeds it back to the main engine. The main engine then generates a response plan based on the error report, such as resending the data, notifying the administrator, and other countermeasures to ensure the accuracy and reliability of data processing.

[0075] After the format conversion is complete, the main engine determines the type of business processing to perform based on the business processing scope of the plug-in mode. Specifically, if the lazy loading mode includes a business sending module, the main engine waits for the lazy loading module to complete data conversion and sending before processing the next batch of data. If the lazy loading mode does not include a business sending module, the main engine receives the converted standard data and sends it through the business sending module.

[0076] Based on the same inventive concept, an embodiment of the present invention provides a data format conversion device based on dynamic lazy loading and applied to a main engine. Figure 2 This is a structural block diagram of a data format conversion device 200 based on dynamic lazy loading provided by an embodiment of the present invention. Figure 2 As shown, the data format conversion device 200 based on dynamic lazy loading mainly includes:

[0077] An acquisition module 201 is configured to acquire original data and target format information in response to a data transmission request from a client;

[0078] The object generation module 202 is used to parse the original data and generate data objects;

[0079] The search and loading module 203 is used to search for a target conversion plug-in that matches the target format information if format conversion is required for the original data, and load the searched target conversion plug-in into the memory;

[0080] The plug-in calling module 204 is used to call the plug-in method through reflection after the target conversion plug-in is loaded, so that the target conversion plug-in obtains the data object, maps and serializes the data object, and obtains the target data to be transmitted.

[0081] In some optional embodiments, the data format conversion device 200 based on dynamic lazy loading further includes:

[0082] The identification module (not shown in the figure) is connected to the acquisition module 201 and the search and load module 203 respectively, and is used to identify whether the original data is valid; if the original data is valid, determine the valid data type; and determine whether the original data needs to be format converted based on the valid data type and target format information.

[0083] In some optional embodiments, the object generation module 202 is specifically used to parse the original data and obtain DML data, DDL data and SQL data; using DML data, DDL data and SQL data as data, the operation statement framework is encapsulated into an operation data object, the data dictionary is encapsulated into a dictionary data object, and the business data is encapsulated into a business data object.

[0084] In some optional embodiments, the data format conversion device 200 based on dynamic lazy loading further includes:

[0085] The reading and loading module (not shown in the figure) is used to read all plug-in jar files in the plug-in directory when the main engine is started, and load the plug-in jar files into the Java virtual machine; each plug-in jar file corresponds to a conversion plug-in, and the plug-in jar file includes data format conversion logic, business processing methods and plug-in functions.

[0086] In some optional embodiments, the data format conversion device 200 based on dynamic lazy loading further includes:

[0087] The plug-in adding module (not shown in the figure) is used to obtain the new plug-in project created by the developer in the integrated development environment when a new customer data conversion customization requirement is added, and to package and deploy the plug-in project to the plug-in directory.

[0088] In some optional embodiments, the data format conversion device 200 based on dynamic lazy loading further includes:

[0089] An initialization module (not shown in the figure) is used to assign unique identification information to each format conversion plug-in when the plug-in library is initialized, and to establish a mapping table between the unique identification information and the target format information;

[0090] The search loading module 203 is specifically configured to search the mapping table for unique identification information that has a mapping relationship with the target format information; and use the format conversion plug-in corresponding to the unique identification information as the target conversion plug-in.

[0091] In some optional embodiments, the data format conversion device 200 based on dynamic lazy loading further includes:

[0092] The verification receiving module (not shown in the figure) is used to receive the target data sent by the target conversion plug-in after the target data verification is passed, and send the target data to the target recipient; or receive the error report information sent by the target conversion plug-in after the target data verification fails, and generate a processing plan based on the error report information.

[0093] The functional modules in the embodiments of the present invention can be integrated together to form an independent unit, for example, integrated into a processing unit, or each module can exist physically separately, or two or more modules can be integrated to form an independent unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a personal computer, server or network device, etc.) or a processor to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk.

[0094] The various variations and specific examples of the method provided in the embodiment of the present invention are also applicable to the data format conversion device based on dynamic lazy loading provided in this embodiment. Through the above detailed description of the data format conversion method based on dynamic lazy loading, those skilled in the art can clearly understand the implementation method of the data format conversion device based on dynamic lazy loading in this embodiment. For the sake of brevity of the specification, it will not be described in detail here.

[0095] Figure 3 FIG. 3 is a structural block diagram of an electronic device 300 provided in an embodiment of the present invention. Figure 3 As shown, the electronic device 300 includes a memory 301 , a processor 302 , and a communication bus 303 ; the memory 301 and the processor 302 are connected via the communication bus 303 .

[0096] The memory 301 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 301 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for at least one function, and instructions for implementing the data format conversion method based on dynamic lazy loading provided in the above embodiment, etc. The data storage area may store data involved in the data format conversion method based on dynamic lazy loading provided in the above embodiment, etc.

[0097] The processor 302 may include one or more processing cores. The processor 302 calls the data stored in the memory 301 by running or executing the instructions, programs, code sets or instruction sets stored in the memory 301, and performs various functions and processes data of the present application. The processor 302 can be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller and a microprocessor. It is understandable that for different devices, the electronic device used to implement the above-mentioned processor 302 function can also be other, and the embodiment of the present invention is not specifically limited.

[0098] The communication bus 303 may include a path for transmitting information between the above components. The communication bus 303 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus 303 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 There is only one double arrow in the diagram, but it does not mean that there is only one bus or one type of bus. Figure 3 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0099] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program that can be loaded by a processor and execute the data format conversion method based on dynamic lazy loading as provided in the above embodiment.

[0100] In this embodiment, a computer-readable storage medium may be a tangible device that holds and stores instructions used by an instruction execution device. The computer-readable storage medium may be, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof. Specifically, the computer-readable storage medium may be a portable computer disk, a hard disk, a USB flash drive, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a lectern random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, an optical disc, a magnetic disk, a mechanical encoding device, or any combination thereof.

[0101] The computer program in this embodiment includes a program for executing Figure 1 The program code for the method shown may include instructions corresponding to the steps of the method provided in the above embodiments. The computer program can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The computer program can be executed entirely on a user's computer or as a standalone software package.

[0102] In the embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0103] The terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0104] The above are merely preferred embodiments of the present application and are not intended to limit the present application. Those skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.< / object> < / object> < / object> < / tablecolumnvo> < / tabblecolumnvo> < / object>

Claims

1. A data format conversion method based on dynamic lazy loading, characterized in that: Applied to the main engine, the method includes: Responding to a data transmission request from a client, obtaining original data and target format information; Parsing the original data to generate a data object; If the format of the original data needs to be converted, a format conversion plug-in matching the target format information is searched in the plug-in library as a target conversion plug-in, and the target conversion plug-in is loaded into the memory; After the target conversion plug-in is loaded, the plug-in method is called through reflection, so that the target conversion plug-in obtains the data object, maps and serializes the data object, and obtains the target data to be transmitted; The method further comprises: Identifying whether the original data is valid; If the original data is valid, determining a valid data type; determining whether format conversion needs to be performed on the original data based on the valid data type and the target format information; The determining whether the format conversion of the original data needs to be performed based on the valid data type and the target format information includes: If the valid data type is incompatible with the target format information, the original data needs to be format converted; The parsing of the original data to generate a data object includes: Parsing the original data to obtain DML data, DDL data, and SQL data; Taking the DML data, the DDL data and the SQL data as data, encapsulating the operation statement framework into an operation data object, encapsulating the data dictionary into a dictionary data object, and encapsulating the business data into a business data object; The DDL data includes a data dictionary of user name, table name, table space, column name, column type, column length, and column precision; the DML data includes detailed data of insert, update, and delete table operations; the SQL includes database operation instructions for DDL change SQL, and the database operation instructions include adding or subtracting columns from a table, modifying column types, and adjusting column length and precision; The method further comprises: When the plug-in library is initialized, unique identification information is assigned to each format conversion plug-in, and a mapping table between the unique identification information and target format information is established; The step of searching the plug-in library for a format conversion plug-in that matches the target format information as the target conversion plug-in includes: Searching the mapping table for the unique identification information that has a mapping relationship with the target format information; The format conversion plug-in corresponding to the unique identification information is used as the target conversion plug-in.

2. The data format conversion method based on dynamic lazy loading according to claim 1, characterized in that: The method further comprises: When the main engine is started, all plug-in jar files in the plug-in directory are read and loaded into the Java virtual machine; each plug-in jar file corresponds to a conversion plug-in, and the plug-in jar file includes data format conversion logic, business processing methods and plug-in functions.

3. The data format conversion method based on dynamic lazy loading according to claim 2, characterized in that: The method further comprises: When a new customer data conversion customization requirement is added, a new plug-in project created by a developer in an integrated development environment is obtained, and the plug-in project is packaged and deployed to the plug-in directory.

4. The data format conversion method based on dynamic lazy loading according to any one of claims 1 to 3, characterized in that: The method further comprises: receiving the target data sent by the target conversion plug-in after verifying the target data, and sending the target data to a target recipient; or Receive error report information sent by the target conversion plug-in after the target data fails to be verified, and generate a processing solution based on the error report information.

5. A data format conversion device based on dynamic lazy loading, characterized in that: Applied to the main engine, the device comprises: An acquisition module, configured to respond to a data transmission request from a client and acquire original data and target format information; An object generation module, configured to parse the original data and generate a data object; A search loading module is used to search for a format conversion plug-in matching the target format information in a plug-in library as a target conversion plug-in if format conversion is required for the original data, and load the target conversion plug-in into memory; A plug-in calling module is used to call the plug-in method through reflection after the target conversion plug-in is loaded, so that the target conversion plug-in obtains the data object, maps and serializes the data object, and obtains the target data to be transmitted; The device further comprises: an identification module, configured to identify whether the original data is valid; if the original data is valid, determine a valid data type; and determine whether the original data needs to be format converted based on the valid data type and the target format information; wherein the determining whether the original data needs to be format converted based on the valid data type and the target format information includes determining that the original data needs to be format converted if the valid data type is incompatible with the target format information; The object generation module is specifically used to parse the original data to obtain DML data, DDL data and SQL data; using the DML data, DDL data and SQL data as data, encapsulate the operation statement framework into an operation data object, encapsulate the data dictionary into a dictionary data object, and encapsulate the business data into a business data object; The DDL data includes a data dictionary of user name, table name, table space, column name, column type, column length, and column precision; the DML data includes detailed data of insert, update, and delete table operations; the SQL includes database operation instructions for DDL change SQL, and the database operation instructions include adding or subtracting columns from a table, modifying column types, and adjusting column length and precision; The device further comprises: An initialization module, configured to assign unique identification information to each format conversion plug-in when the plug-in library is initialized, and to establish a mapping table between the unique identification information and target format information; The search loading module is specifically configured to search the mapping table for the unique identification information that has a mapping relationship with the target format information; and use the format conversion plug-in corresponding to the unique identification information as the target conversion plug-in.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the data format conversion method based on dynamic lazy loading according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data format conversion method based on dynamic lazy loading according to any one of claims 1 to 4 is implemented.

Images