Data format conversion method, system, device and storage medium

Abstract

The present invention relates to the field of data processing technology, and specifically provides a data format conversion method, system, device and storage medium, including: intercepting a data transmission request at the front end, extracting the data source, original data format, data destination address, data name and data from the data transmission request; inputting the data source, original data format, data destination address and data name into a pre-trained prediction model to predict the data usage format; confirming that the data usage format is inconsistent with the original data format, generating a format conversion task for the data and caching the format conversion task to a task queue, the format conversion task including converting the data from the original data format to the data usage format; calling a format converter to execute the format conversion task, and sending the data after the format conversion is completed to the data destination address. The present invention effectively reduces the processing burden of the receiving end and improves the efficiency and accuracy of data transmission.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and specifically relates to a data format conversion method, system, device and storage medium. Background Art

[0002] In today's highly information-based society, data, as an important resource, plays a vital role in all walks of life. With the continuous advancement of technology, the sources and formats of data are becoming increasingly diverse, and different systems and platforms need to exchange and share data frequently. However, due to the diversity and incompatibility of data formats, data often faces the problem of format conversion during transmission and exchange. At present, most data transmission and exchange scenarios adopt the method of format conversion at the data receiving end. That is, after the data is sent from the sending end and reaches the receiving end, the receiving end converts the data format as needed to meet the requirements of its system or application. However, this method has several obvious problems: First, the data receiving end needs to bear all the format conversion work, which not only increases the processing burden of the receiving end, but also may cause processing delays, affecting the real-time and availability of data. Especially in the case of large data volume and high concurrency, the processing pressure of the receiving end will increase exponentially, and may even cause system congestion or crash. Secondly, due to the diversity and complexity of data formats, the receiving end often needs to develop corresponding conversion tools or modules for different data formats, which not only increases the development cost and maintenance difficulty, but also may cause data loss, damage or format confusion due to imperfections or errors in the conversion tools. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention provides a data format conversion method, system, device and storage medium to solve the above-mentioned technical problems.

[0004] In a first aspect, the present invention provides a data format conversion method, comprising:

[0005] intercepting data transmission requests at the front end, and extracting data source, original data format, data destination address, data name and data from the data transmission requests;

[0006] Input the data source, original data format, data destination address, and data name into the pre-trained prediction model to predict the data usage format;

[0007] Confirming that the data usage format is inconsistent with the original data format, generating a format conversion task for the data and caching the format conversion task to a task queue, wherein the format conversion task includes converting the data from the original data format to the data usage format;

[0008] The format converter is called to execute the format conversion task, and the data after the format conversion is completed is sent to the data destination address.

[0009] In an optional embodiment, the method further comprises:

[0010] Collecting front-end historical behavior data, the historical behavior data includes behavior data within a specified historical period, the behavior data includes data source, original data format, data destination address, data usage format, data name and behavior execution time;

[0011] The historical period is evenly divided into three time periods, and the behavior data with the same behavior execution time in the same time period are divided into the same data set to obtain three data sets corresponding to the three time periods;

[0012] Perform cluster analysis on the behavioral data in the three data sets respectively, and output the centroid of each cluster in the clustering results as feature data;

[0013] Establishing a correspondence rule between the data usage format and the data source, the original data format, the data destination address, and the data name based on the characteristic data;

[0014] The similarity of the corresponding rules of the three data sets is calculated. If the similarity reaches the set threshold, all corresponding rules are integrated into a prediction model; if the similarity does not reach the set threshold, the corresponding rules of the data set with the time period closest to the current time are integrated into the prediction model.

[0015] In an optional implementation, cluster analysis is performed on the behavior data in the three data sets respectively, and the centroid of each cluster in the clustering result is output as feature data, including:

[0016] The data of the three data sets were cleaned separately to remove duplicate data and process missing values, outliers and noise data;

[0017] The behavioral data are quantified to convert them into standard behavioral data.

[0018] In an optional implementation, cluster analysis is performed on the behavior data in the three data sets respectively, and the centroid of each cluster in the clustering result is output as feature data, including:

[0019] Count the number of statistical data formats and set the number as the K value of the K-means algorithm;

[0020] The silhouette coefficient and the Rand index are calculated based on the clustering results. If the silhouette coefficient and the Rand index reach the set threshold, the clustering results are adopted; if the silhouette coefficient and the Rand index do not reach the set threshold, the K value is adjusted and clustering is re-executed.

[0021] In an optional implementation, calling a format converter to perform the format conversion task and sending the data after the format conversion is completed to the data destination address includes:

[0022] Replacing the data in the data transmission request with data whose format has been converted;

[0023] The data transmission request is executed to send the data after format conversion to the data destination address.

[0024] In an optional embodiment, the method further comprises:

[0025] Confirm that the data usage format is consistent with the original data format, and directly execute the data transmission request.

[0026] In a second aspect, the present invention provides a data format conversion system, comprising:

[0027] A request interception module is used to intercept data transmission requests at the front end, and extract data source, original data format, data destination address, data name and data from the data transmission request;

[0028] The format prediction module is used to input the data source, original data format, data destination address, and data name into the pre-trained prediction model to predict the data usage format;

[0029] A format conversion module, used to confirm that the data usage format is inconsistent with the original data format, generate a format conversion task for the data and cache the format conversion task to a task queue, wherein the format conversion task includes converting the data from the original data format to the data usage format;

[0030] The request execution module is used to call the format converter to execute the format conversion task and send the data after the format conversion is completed to the data destination address.

[0031] In a third aspect, a device is provided, comprising:

[0032] A memory, used for storing a data format conversion program;

[0033] A processor is used to implement the steps of the data format conversion method provided in the first aspect when executing the data format conversion program.

[0034] In a fourth aspect, a computer-readable storage medium is provided, on which a data format conversion program is stored. When the data format conversion program is executed by a processor, the steps of the data format conversion method provided in the first aspect are implemented.

[0035] The beneficial effect of the present invention is that the data format conversion method, system, device and storage medium provided by the present invention predict the usage format of data through a pre-trained prediction model, and generate a format conversion task for conversion when it is confirmed that the data usage format is inconsistent with the original data format. This method moves the work of format conversion to the initial stage of data transmission, thereby effectively reducing the processing burden on the receiving end and improving the efficiency and accuracy of data transmission. At the same time, through intelligent prediction and automatic conversion, the present invention can also adapt to the data format requirements of different systems and platforms, and realize seamless docking and efficient sharing of data.

[0036] In addition, the invention has a reliable design principle, a simple structure and a very broad application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention.

[0039] Figure 2 is a schematic block diagram of a system according to an embodiment of the present invention.

[0040] Figure 3 A schematic diagram of the structure of a device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0043] The data format conversion method provided in the embodiment of the present invention is executed by a computer device, and accordingly, the data format conversion system runs in the computer device.

[0044] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention. Figure 1 The execution subject may be a data format conversion system. According to different requirements, the order of the steps in the flow chart may be changed, and some may be omitted.

[0045] like Figure 1 As shown, the method includes:

[0046] S1. intercepting the data transmission request at the front end, and extracting the data source, original data format, data destination address, data name and data from the data transmission request.

[0047] In the front-end system, interception of data transmission requests is the first step in the implementation of the present invention. To achieve this function, we deploy a request interceptor that can capture all data transmission requests issued through the front-end. These requests may originally be triggered by user operations, system events, or other applications.

[0048] Once the request is captured by the interceptor, the system will immediately enter the data processing process. First, the interceptor will extract key information from the data transmission request, including but not limited to the data source, original data format, data destination address, data name, and actual data content. Extracting this information is the basis for subsequent steps and ensures the accuracy and efficiency of data processing.

[0049] S2. Input the data source, original data format, data destination address, and data name into the pre-trained prediction model to predict the data usage format.

[0050] The extracted data source, original data format, data destination address and data name are passed as input to a pre-trained prediction model. This prediction model is built based on a machine learning algorithm. It can predict the data usage format corresponding to the current data request by analyzing historical data conversion records and learning the association rules between data formats. The output of the prediction model is one or more possible data usage formats, which are sorted from high to low according to the prediction probability.

[0051] S3. Confirm that the data usage format is inconsistent with the original data format, generate a format conversion task for the data and cache the format conversion task to a task queue, wherein the format conversion task includes converting the data from the original data format to the data usage format.

[0052] After the system receives the data usage format output by the prediction model, it will compare it with the original data format. If the predicted data usage format is inconsistent with the original data format, it means that the data needs to be format converted to meet the usage requirements of the destination address. At this time, the system will generate a format conversion task for this data. Each format conversion task contains detailed conversion information, such as original data, original data format, target data format, etc.

[0053] After the format conversion tasks are generated, they are cached in a task queue. The task queue adopts a first-in-first-out (FIFO) management strategy to ensure that the format conversion tasks can be processed in the order in which they are received. At the same time, the task queue also has the ability of load balancing and fault tolerance. It can dynamically adjust the processing speed of tasks according to the actual load of the system, and automatically retry or rollback operations when errors occur.

[0054] S4. Calling the format converter to execute the format conversion task, and sending the data after the format conversion is completed to the data destination address.

[0055] The conversion tasks are taken out from the task queue in sequence and executed, and after the format conversion is completed, the data whose format conversion is completed is sent to the data destination address.

[0056] In an embodiment of the present invention, based on step S1, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0057] S101. A user operates through the front-end interface, or a data transmission request is triggered by a system event or other application.

[0058] S102. The request interceptor deployed in the front-end system captures all data transmission requests issued through these channels.

[0059] S103. The interceptor starts processing the captured request and first extracts the following key information:

[0060] Data Origin: Determine the origin of the data, such as which user, which system, or which application.

[0061] Original data format: Understand the original format of the data, such as JSON, XML, text, etc.

[0062] Data destination address: Determines the target address or system to which the data should be sent.

[0063] Data Name: Assign a unique name or identifier to the data.

[0064] Actual data content: Get the actual data content carried in the request.

[0065] In an embodiment of the present invention, based on step S2, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0066] First, a prediction model needs to be constructed. In this embodiment, the prediction model is constructed as follows:

[0067] (1) Collecting front-end historical behavior data, wherein the historical behavior data includes behavior data within a specified historical period, and the behavior data includes data source, original data format, data destination address, data usage format, data name and behavior execution time.

[0068] It is necessary to clarify the source of the data, including user interaction data, system logs, application logs, etc. These data sources provide rich behavioral data, including user clicks, browsing, purchases, searches, and other behaviors.

[0069] Before collecting data, define the data fields to be collected, including data source, original data format, data destination address, data usage format, data name, and behavior execution time, etc. These fields are the basis for subsequent data processing and analysis.

[0070] Data collection can be achieved in many ways, such as using front-end log collection tools, back-end API records, etc. Ensuring the accuracy and completeness of the data is key in this step.

[0071] (2) Divide the historical period into three time periods on average, and divide the behavior data with the same execution time in the same time period into the same data set, thereby obtaining three data sets corresponding to the three time periods.

[0072] Time period definition: Divide the specified historical period into three time periods, which helps us observe and analyze the changing trends of data in different time periods. The division of time periods can be flexibly adjusted according to business needs, such as by quarter, month or week.

[0073] Data allocation: According to the execution time of the behavior, each behavior data is allocated to the corresponding time period data set. This step ensures that each data set contains the behavior data within a specific time period.

[0074] (3) The data of the three data sets were cleaned separately to remove duplicate data, and to process missing values, outliers, and noise data; the behavioral data were quantified to convert them into standard behavioral data.

[0075] Duplicate data identification: Use unique identifiers (such as user ID, session ID, etc.) to identify and remove duplicate data records.

[0076] Missing value processing: For missing data fields, we can use methods such as filling default values, filling with the previous or next value, and interpolation to handle them.

[0077] Outlier detection and processing: Identify outliers through statistical methods or machine learning algorithms and take appropriate processing measures, such as deletion, replacement or smoothing.

[0078] Noise data filtering: For data containing noise, we can use filtering techniques or data smoothing methods to reduce the impact of noise on data analysis.

[0079] In a specific example, the behavior data quantification method is as follows:

[0080] Data source: Although the data source itself may be a complex string (such as a URL, IP address, or system name), we can treat it as a categorical variable and assign a unique quantitative value to each different source. For example, source A can be quantified as V1, source B as V2, and so on.

[0081] Original data format: Based on common data formats, we can assign a quantification value to each format. For example, XLM format can be quantified as G1, JSON format can be quantified as G2, CSV format can be quantified as G3, text format can be quantified as G4, etc.

[0082] Data Destination Address: Similar to the data source, the data destination address can also be treated as a categorical variable and assigned a unique quantitative value.

[0083] Data usage format: This is usually related to the original data format, but can also be different (e.g., the original data might be XML, but the usage format might be JSON). Therefore, we can also assign a quantitative value to each usage format.

[0084] Data name quantification: Data names usually describe the specific content or purpose of the data. We can parse the data type based on the data name (for example, whether it is intermediate data, initial data, user behavior data, etc.), and assign a unique quantification value to each type. For example, initial data can be quantified as T1, intermediate data can be quantified as T2, user behavior data can be quantified as T3, and so on. When parsing the data type, we can use the keywords or contextual information of the data name to determine its type. For example, if the data name contains words such as "log" or "record", then it may belong to intermediate data; if the data name is directly related to a specific business scenario (such as "purchase record", "browsing record", etc.), it may belong to user behavior data.

[0085] (4) Perform cluster analysis on the behavioral data in the three data sets respectively, and output the characteristic data of the centroid of each cluster in the clustering results; count the number of statistical formats and set the number as the K value of the K-means algorithm; calculate the silhouette coefficient and the Rand index based on the clustering results, and if the silhouette coefficient and the Rand index reach the set threshold, adopt the clustering results; if the silhouette coefficient and the Rand index do not reach the set threshold, adjust the K value and re-execute clustering.

[0086] During data preprocessing, count the number of different data formats in each dataset. For example, dataset 1 may have three formats: JSON, XML, and CSV, dataset 2 may have only two formats: JSON and CSV, and dataset 3 may have four formats: JSON, XML, CSV, and text.

[0087] The total number of different data formats in all datasets is used as the K value of the K-means algorithm. In this example, the K value is 4 (JSON, XML, CSV, text).

[0088] The K-means algorithm is used to perform cluster analysis on the three data sets and four clusters are obtained (assuming that the K value is selected reasonably).

[0089] Calculate the centroid of each cluster as the characteristic data of the cluster. The centroid is the average value of all points in the cluster and represents the center position of the cluster.

[0090] The silhouette coefficient and Rand index are calculated to evaluate the clustering effect. The silhouette coefficient measures the closeness of the data point to its cluster and the degree of separation from other clusters. The value range is [-1, 1]. The larger the value, the better the clustering effect. The Rand index measures the consistency between the clustering result and the true label (in this example, since there is no true label, it can be regarded as an indicator for evaluating the internal consistency of the clustering result). The value range is [0, 1]. The larger the value, the more stable the clustering result.

[0091] If the silhouette coefficient and the Rand index reach the set threshold (such as silhouette coefficient ≥ 0.5, Rand index ≥ 0.7), the clustering result is considered acceptable; otherwise, the K value is increased and the clustering analysis is performed again.

[0092] Suppose in the cluster analysis of data set 1, we obtained the following centroid feature data of the four clusters:

[0093] Cluster 1 (JSON format): platform 1, XML format, platform 2, intermediate data;

[0094] Cluster 2 (XML format): Platform 1, XML format, Platform 3, initial data;

[0095] Cluster 3 (CSV format): Platform 2, JSON format, Platform 4, initial data;

[0096] Cluster 4 (text format): Platform 4, CSV format, front end, output data.

[0097] (5) Based on the characteristic data, a correspondence rule is established between the data usage format and the data source, the original data format, the data destination address, and the data name.

[0098] Based on the above centroid feature data (although hypothetical), we can try to establish the following corresponding rules:

[0099] When the data format is JSON:

[0100] The data may originate from platform 1 or platform 2.

[0101] The raw data formats may include XML (if intra-cluster diversity is a concern) or primarily JSON itself.

[0102] The data may be used in an intermediate data processing system.

[0103] The data name / type may be intermediate data.

[0104] When the data format is XML:

[0105] The data mainly comes from platform 1 and may also come from platform 3.

[0106] The original data format is clearly XML.

[0107] The data may be used in an initial data processing system.

[0108] The data name / type may be original data.

[0109] When the data format is CSV:

[0110] The data may originate from platform 2 or platform 4.

[0111] The raw data formats may include JSON (if intra-cluster diversity is considered) or mainly CSV itself (but in the description of cluster 4 CSV is associated with text format, so be careful here).

[0112] The destination of the data may be unclear or involve multiple systems.

[0113] The data name / type may be the original data (but may also be different depending on the specific situation).

[0114] When the data format is text:

[0115] The data mainly comes from platform 4.

[0116] The original data format may include CSV (in some dimension), but more specific analysis is needed to determine this.

[0117] The data is used for the front-end output system.

[0118] The data name / type may be output data.

[0119] (6) Calculate the similarity of the corresponding rules of the three data sets. If the similarity reaches the set threshold, all corresponding rules are integrated into the prediction model. If the similarity does not reach the set threshold, the corresponding rules of the data set with the time period closest to the current time are integrated into the prediction model.

[0120] Calculate the similarity of corresponding rules:

[0121] First, the corresponding rules for each data set need to be expressed in a structured way. This usually involves breaking down the rules into comparable components, such as data source, original data format, data destination address, data usage format, and data name.

[0122] Choose an appropriate similarity measurement method to compare the corresponding rules of different data sets. Commonly used methods include cosine similarity, Jaccard similarity, edit distance, etc. These methods can be adjusted and optimized according to the specific content and structure of the rules.

[0123] Set a similarity threshold to determine whether the corresponding rules of different data sets are similar enough to be integrated into a unified prediction model. This threshold should be set according to the specific application scenario and data characteristics to ensure the accuracy and reliability of the model.

[0124] Integrate the corresponding rules into a prediction model:

[0125] If the similarity of the corresponding rules of the three data sets reaches or exceeds the set threshold, then these rules can be integrated into a unified prediction model. This usually involves merging, deduplicating and normalizing the rules of different data sets to ensure the consistency and accuracy of the model.

[0126] If the similarity does not reach the set threshold, the corresponding rules of the data set closest to the current time are integrated into the prediction model. This is because the data in the most recent period usually better reflects the current market trends and user behavior patterns, and therefore it is more likely to build an effective prediction model.

[0127] After integrating the corresponding rules, the prediction model needs to be verified and optimized. This usually involves using an independent test data set to evaluate the performance of the model and adjusting and improving the model based on the evaluation results.

[0128] In an embodiment of the present invention, based on step S4, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0129] A data format converter is able to convert one data format to another to meet the needs of different devices, applications or platforms. It supports conversion between multiple file formats, including but not limited to images, audio, video, and documents. Through conversion, users can easily adapt files to different scenarios, such as reducing image file size, converting audio to a more universal format, sharing videos across platforms, and converting document formats in an office environment to keep content secure. The working principle of a data format converter usually involves parsing, processing, and reconstructing the input data. Specifically, it first reads the raw data of the input file, and then processes the data accordingly according to the requirements of the target format, such as decoding, encoding, compression, decompression, etc. Finally, it reorganizes the processed data into a file in the target format and outputs it to the user.

[0130] When there are many data formats involved, you can deploy multiple data format converters to cover the format conversion requirements. When executing a data format conversion task, just call the corresponding converter.

[0131] Replace the format-converted data into the previously generated data transfer request. This step ensures that the request carries the format-converted data instead of the original data.

[0132] The updated data transfer request now contains the data in the correct format and is ready for transfer.

[0133] Finally, the updated data transfer request is executed. This step involves transferring the data from the source address to the destination address, which may be done over a network, file system, or other transmission medium.

[0134] During the transmission process, it is necessary to ensure the integrity and security of the data to avoid data loss or tampering.

[0135] In addition, in one embodiment of the present invention, when the predicted data usage format is consistent with the original data format, the data transmission request is directly executed without generating a data format conversion task.

[0136] In some embodiments, the data format conversion system may include multiple functional modules composed of computer program segments. The computer programs of each program segment in the data format conversion system may be stored in a memory of a computer device and executed by at least one processor to perform (see Figure 1 Description) Data format conversion function.

[0137] In this embodiment, the data format conversion system can be divided into multiple functional modules according to the functions it performs, such as Figure 2 As shown. The functional modules of the system 200 may include: a request interception module 210, a format prediction module 220, a format conversion module 230, and a request execution module 240. The module referred to in the present invention refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0138] A request interception module is used to intercept data transmission requests at the front end, and extract data source, original data format, data destination address, data name and data from the data transmission request;

[0139] The format prediction module is used to input the data source, original data format, data destination address, and data name into the pre-trained prediction model to predict the data usage format;

[0140] A format conversion module, used to confirm that the data usage format is inconsistent with the original data format, generate a format conversion task for the data and cache the format conversion task to a task queue, wherein the format conversion task includes converting the data from the original data format to the data usage format;

[0141] The request execution module is used to call the format converter to execute the format conversion task and send the data after the format conversion is completed to the data destination address.

[0142] Optionally, as an embodiment of the present invention, it further includes:

[0143] Collecting front-end historical behavior data, the historical behavior data includes behavior data within a specified historical period, the behavior data includes data source, original data format, data destination address, data usage format, data name and behavior execution time;

[0144] The historical period is evenly divided into three time periods, and the behavior data with the same behavior execution time in the same time period are divided into the same data set to obtain three data sets corresponding to the three time periods;

[0145] Perform cluster analysis on the behavioral data in the three data sets respectively, and output the centroid of each cluster in the clustering results as feature data;

[0146] Establishing a correspondence rule between the data usage format and the data source, the original data format, the data destination address, and the data name based on the characteristic data;

[0147] The similarity of the corresponding rules of the three data sets is calculated. If the similarity reaches the set threshold, all corresponding rules are integrated into a prediction model; if the similarity does not reach the set threshold, the corresponding rules of the data set with the time period closest to the current time are integrated into the prediction model.

[0148] Optionally, as an embodiment of the present invention, cluster analysis is performed on the behavior data in the three data sets respectively, and the centroid of each cluster in the clustering result is output as feature data, including:

[0149] The data of the three data sets were cleaned separately to remove duplicate data and process missing values, outliers and noise data;

[0150] The behavioral data are quantified to convert them into standard behavioral data.

[0151] Optionally, as an embodiment of the present invention, cluster analysis is performed on the behavior data in the three data sets respectively, and the centroid of each cluster in the clustering result is output as feature data, including:

[0152] Count the number of statistical data formats and set the number as the K value of the K-means algorithm;

[0153] The silhouette coefficient and the Rand index are calculated based on the clustering results. If the silhouette coefficient and the Rand index reach the set threshold, the clustering results are adopted; if the silhouette coefficient and the Rand index do not reach the set threshold, the K value is adjusted and clustering is re-executed.

[0154] Optionally, as an embodiment of the present invention, calling a format converter to perform the format conversion task and sending the data after the format conversion is completed to the data destination address includes:

[0155] Replacing the data in the data transmission request with data whose format has been converted;

[0156] The data transmission request is executed to send the data after format conversion to the data destination address.

[0157] Optionally, as an embodiment of the present invention, it further includes:

[0158] Confirm that the data usage format is consistent with the original data format and directly execute the data transmission request

[0159] Figure 3The data format conversion method provided for the embodiment of the present application can be applied to equipment. It will be appreciated by those skilled in the art that the equipment structure involved in the embodiment of the present invention does not constitute a limitation on the equipment, and the equipment may include more or less components than shown, or combine certain components, or different component arrangements. In an embodiment of the present invention, the equipment includes but is not limited to laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The equipment may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the embodiments of the present application described herein and / or required.

[0160] The device 300 may include: a processor 310, a memory 320 and a communication unit 330. These components communicate via one or more buses. Those skilled in the art will appreciate that the server structure shown in the figure does not limit the present invention, and it may be a bus structure or a star structure, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0161] The memory 320 may be used to store the execution instructions of the processor 310, and the memory 320 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the device 300 is enabled to perform some or all of the steps in the following method embodiments.

[0162] The processor 310 is the control center of the storage device, and uses various interfaces and lines to connect various parts of the entire electronic device. It runs or executes software programs and / or modules stored in the memory 320, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of a plurality of packaged ICs with the same or different functions. For example, the processor 310 can include only a central processing unit (CPU). In an embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.

[0163] The communication unit 330 is used to establish a communication channel so that the storage device can communicate with other devices, receive user data sent by other devices or send user data to other devices.

[0164] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program may include some or all of the steps in each embodiment provided by the present invention. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0165] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes, including several instructions for enabling a computer device (which can be a personal computer, a server, or a second device, a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0166] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.

[0167] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or modules, which can be electrical, mechanical or other forms.

[0168] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0170] Although the present invention has been described in detail with reference to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions shall be within the scope of the present invention. Any person of ordinary skill in the art may easily think of changes or substitutions within the technical scope disclosed by the present invention, and these shall be within the scope of protection of the present invention.

Claims

1. A data format conversion method, characterized in that: include: intercepting data transmission requests at the front end, and extracting data source, original data format, data destination address, data name and data from the data transmission requests; Input the data source, original data format, data destination address, and data name into the pre-trained prediction model to predict the data usage format; Confirming that the data usage format is inconsistent with the original data format, generating a format conversion task for the data and caching the format conversion task to a task queue, wherein the format conversion task includes converting the data from the original data format to the data usage format; Calling a format converter to execute the format conversion task, and sending the data after the format conversion is completed to the data destination address; The method further comprises: Collecting front-end historical behavior data, the historical behavior data includes behavior data within a specified historical period, the behavior data includes data source, original data format, data destination address, data usage format, data name and behavior execution time; The historical period is evenly divided into three time periods, and the behavior data with the same behavior execution time in the same time period are divided into the same data set to obtain three data sets corresponding to the three time periods; Perform cluster analysis on the behavioral data in the three data sets respectively, and output the centroid of each cluster in the clustering results as feature data; Establishing a correspondence rule between the data usage format and the data source, the original data format, the data destination address, and the data name based on the characteristic data; The similarity of the corresponding rules of the three data sets is calculated. If the similarity reaches the set threshold, all corresponding rules are integrated into a prediction model; if the similarity does not reach the set threshold, the corresponding rules of the data set with the time period closest to the current time are integrated into the prediction model.

2. The method according to claim 1, characterized in that Perform cluster analysis on the behavior data in the three data sets respectively, and output the characteristic data of the centroid of each cluster in the clustering results, including: The data of the three data sets were cleaned separately to remove duplicate data and process missing values, outliers and noise data; The behavioral data are quantified to convert them into standard behavioral data.

3. The method according to claim 1, characterized in that Perform cluster analysis on the behavior data in the three data sets respectively, and output the characteristic data of the centroid of each cluster in the clustering results, including: Count the number of statistical data formats and set the number as the K value of the K-means algorithm; The silhouette coefficient and the Rand index are calculated based on the clustering results. If the silhouette coefficient and the Rand index reach the set threshold, the clustering results are adopted; if the silhouette coefficient and the Rand index do not reach the set threshold, the K value is adjusted and clustering is re-executed.

4. The method according to claim 1, characterized in that: Calling a format converter to perform the format conversion task and sending the data after the format conversion is completed to the data destination address includes: Replacing the data in the data transmission request with data whose format conversion has been completed; The data transmission request is executed to send the data after format conversion to the data destination address.

5. The method according to claim 1, characterized in that The method further comprises: Confirm that the data usage format is consistent with the original data format, and directly execute the data transmission request.

6. A data format conversion system, characterized in that: include: A request interception module is used to intercept data transmission requests at the front end, and extract data source, original data format, data destination address, data name and data from the data transmission request; The format prediction module is used to input the data source, original data format, data destination address, and data name into the pre-trained prediction model to predict the data usage format; A format conversion module, used to confirm that the data usage format is inconsistent with the original data format, generate a format conversion task for the data and cache the format conversion task to a task queue, wherein the format conversion task includes converting the data from the original data format to the data usage format; A request execution module, used for calling the format converter to execute the format conversion task, and sending the data after the format conversion is completed to the data destination address; The system further comprises: A data collection module is used to collect front-end historical behavior data, wherein the historical behavior data includes behavior data within a specified historical period, and the behavior data includes data source, original data format, data destination address, data usage format, data name and behavior execution time; A data partitioning module, used to divide the historical period into three time periods on average, and divide the behavior data with behavior execution time in the same time period into the same data set, so as to obtain three data sets corresponding to the three time periods; The clustering analysis module is used to perform cluster analysis on the behavior data in the three data sets respectively, and output the centroid of each cluster in the clustering results as feature data; A rule generation module, used to establish a corresponding rule between the data usage format and the data source, the original data format, the data destination address, and the data name based on the characteristic data; The rule integration module is used to calculate the similarity of the corresponding rules of the three data sets. If the similarity reaches the set threshold, all the corresponding rules are integrated into the prediction model; if the similarity does not reach the set threshold, the corresponding rules of the data set with the time period closest to the current time are integrated into the prediction model.

7. A data format conversion device, characterized in that: include: A memory, used for storing a data format conversion program; A processor, used to implement the steps of the data format conversion method as described in any one of claims 1 to 5 when executing the data format conversion program.

8. A computer-readable storage medium storing a computer program, characterized in that: The readable storage medium stores a data format conversion program, and when the data format conversion program is executed by the processor, the steps of the data format conversion method according to any one of claims 1 to 5 are implemented.

Images