Cross-platform log data synchronization method and device, equipment and storage medium

Abstract

The invention discloses a cross-platform log data synchronization method and device, equipment and a storage medium. Comprising the following steps: respectively acquiring log data from platforms on two sides, and determining field features of the log data; the log data of the platforms on the two sides are matched based on the field features to generate a total matching value, and a mapping relation is generated according to the total matching value; and synchronizing the log data of the platforms on the two sides according to the mapping relationship. Data matching can be carried out more accurately by respectively acquiring the field features, including data types, value ranges and distribution features, of the log data of the platforms on the two sides, so that the accuracy of data synchronization is greatly improved, the conditions of data errors and inconsistency are reduced, and the matching and mapping method based on the field features can be applied to the field matching of the log data of the platforms on the two sides. The method does not depend on a specific platform or data format, can adapt to various log data of different types and structures, and has very high universality and expandability. Effective synchronization of log data of platforms on two sides is realized, and a user can conveniently share data among different platforms.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a cross-platform log data synchronization method, device, equipment and storage medium. Background Art

[0002] In today's digital age, people often need to use a variety of different types of devices and operating systems in their work and life, such as Windows, Mac OS, Android, iOS, etc. Different platforms differ in data storage, format and transmission methods.

[0003] With the rapid development of information technology, the amount of data generated and used by users on different platforms has increased dramatically. However, due to the inherent differences and technical limitations between platforms, data synchronization between different platforms faces many challenges.

[0004] Traditional data synchronization methods often have many shortcomings. For example, some methods may only support specific operating systems or applications, resulting in limited applicability. Some synchronization methods may have defects in ensuring data integrity and consistency, and are prone to data loss, duplication, or inconsistency. In addition, some synchronization solutions are inefficient when processing large amounts of data, resulting in long synchronization times and affecting the user experience. Summary of the invention

[0005] The present invention provides a cross-platform log data synchronization method, device, equipment and storage medium to achieve data synchronization between platforms on both sides, and monitor the data synchronization process to ensure the stability of data synchronization.

[0006] According to one aspect of the present invention, a cross-platform log data synchronization method is provided, the method comprising:

[0007] Obtain log data from both platforms and determine the field characteristics of the log data, where the field characteristics include data type, value range, and distribution characteristics;

[0008] Match the log data of both platforms based on field features to generate a total matching value, and generate a mapping relationship based on the total matching value;

[0009] The log data of both platforms are synchronized according to the mapping relationship.

[0010] Optionally, log data is obtained from both platforms, including: collecting call logs from both platforms, wherein the call logs include request and response parameters, call timestamps, and call results; performing data cleansing on the call logs to generate cleaned call logs; and performing data standardization on the cleaned call logs to generate log data.

[0011] Optionally, determining the field characteristics of the log data includes: determining the data type and field value of each field in the log data; determining the minimum and maximum values ​​of each field based on the field value, and generating a value range for each field based on the minimum and maximum values; analyzing the frequency distribution of the field values ​​to determine the distribution characteristics of each field; and using the data type, value range and distribution characteristics as field characteristics.

[0012] Optionally, the log data of the platforms on both sides are matched based on field characteristics to generate a total matching value, including: matching the fields of the platforms on both sides based on field characteristics to determine the matching degree of each field pair, wherein the matching degree includes data type matching degree, value range matching degree and distribution characteristic matching degree; obtaining weight parameters corresponding to the characteristics of each field, and combining the weight parameters and the corresponding matching degree to generate a total matching value for each field pair.

[0013] Optionally, a mapping relationship is generated based on the total matching value, including: when the total matching value is greater than a preset threshold, the field pair is used as a target field pair; and a mapping relationship between the platform fields on both sides is generated based on the corresponding relationship of the target field pair.

[0014] Optionally, the two platforms include a first platform and a second platform, and log data of the two platforms are synchronized according to a mapping relationship, including: determining the fields to be synchronized corresponding to the first platform according to the mapping relationship; converting the fields to be synchronized based on the mapping relationship to generate conversion data; and storing the conversion data on the second platform.

[0015] Optionally, the method also includes: monitoring the data synchronization process based on preset monitoring indicators; when an abnormal situation is monitored, determining the type of abnormality, and determining adjustment parameters based on the abnormality type, wherein the abnormality types include data type mismatch, field value out of range, field missing, and inconsistent nested structure; adjusting the mapping relationship based on the adjustment parameters.

[0016] According to another aspect of the present invention, a cross-platform log data synchronization device is provided, the device comprising:

[0017] The log data acquisition module is used to obtain log data from the two platforms and determine the field characteristics of the log data, wherein the field characteristics include data type, value range and distribution characteristics;

[0018] The log data matching module is used to match the log data of the two platforms based on field features to generate a total matching value, and generate a mapping relationship based on the total matching value;

[0019] The log data synchronization module is used to synchronize the log data of the two platforms according to the mapping relationship.

[0020] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0021] at least one processor;

[0022] and a memory communicatively coupled to the at least one processor;

[0023] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a cross-platform log data synchronization method described in any embodiment of the present invention.

[0024] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a cross-platform log data synchronization method described in any embodiment of the present invention when executed.

[0025] The technical solution of the embodiment of the present invention can more accurately match data by respectively obtaining the field characteristics of the log data of the platforms on both sides, including data type, value range and distribution characteristics, thereby greatly improving the accuracy of data synchronization and reducing data errors and inconsistencies. The matching and mapping method based on field characteristics does not rely on a specific platform or data format, can adapt to various types and structures of log data, and has strong versatility and scalability. Effective synchronization of log data on the platforms on both sides is achieved, making it convenient for users to share data between different platforms.

[0026] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 is a flowchart of a cross-platform log data synchronization method provided according to Embodiment 1 of the present invention;

[0029] Figure 2 is a flowchart of another cross-platform log data synchronization method provided according to Embodiment 2 of the present invention;

[0030] Figure 3 It is a schematic diagram of the structure of a cross-platform log data synchronization device provided according to Embodiment 3 of the present invention;

[0031] Figure 4 It is a structural schematic diagram of an electronic device for implementing a cross-platform log data synchronization method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] Embodiment 1

[0035] Figure 1 A flowchart of a cross-platform log data synchronization method is provided for the first embodiment of the present invention. This embodiment is applicable to the case of data synchronization between two platforms. The method can be executed by a cross-platform log data synchronization device. The cross-platform log data synchronization device can be implemented in the form of hardware and / or software. The cross-platform log data synchronization device can be configured in a computer controller. Figure 1 As shown, the method includes:

[0036] S110, respectively obtaining log data from the platforms on both sides, and determining field characteristics of the log data, wherein the field characteristics include data type, value range, and distribution characteristics.

[0037] The two platforms refer to the operating environments or data storage areas corresponding to two different systems, software or applications. The two platforms generate and store log data respectively, and may have different architectures, functions and data formats. Field characteristics include data types, value ranges and distribution characteristics. Data types refer to the types of data stored in each field in the log data, including strings (text data composed of characters), integers (numbers without decimal parts), floating point numbers (numbers with decimal parts), dates and times (indicating specific dates and time points), etc. The value range refers to the range of values ​​that a field in the log data may take. For example, a field representing the age of the user may have a value range between 0 and 120 years old. Distribution characteristics refer to the distribution of the values ​​of a field in the log data in the entire data set. For example, the values ​​of a field may show a normal distribution (bell curve), uniform distribution or skewed distribution.

[0038] Optionally, log data is obtained from both platforms, including: collecting call logs from both platforms, wherein the call logs include request and response parameters, call timestamps, and call results; performing data cleansing on the call logs to generate cleaned call logs; and performing data standardization on the cleaned call logs to generate log data.

[0039] The request and response parameters include field name, data type, value, etc. The call timestamp records the time when the API is called. The call result indicates the final status of the call, usually success or failure.

[0040] Specifically, there may be some irrelevant, erroneous or duplicate data in the call log, i.e., noise data. Data cleaning is to remove noise data to improve the quality of the data. For example, you can set filters to remove some specific request parameters, such as debugging information, log level, etc. You can also remove some duplicate call logs by comparing timestamps and call results to avoid data redundancy. In addition, there may be some erroneous data in the call log, such as format errors, missing data, etc. The data cleaning process will correct the erroneous data to ensure the integrity and accuracy of the data. For example, if the format of a request parameter is incorrect, you can use regular expressions and other methods to perform format verification and automatically correct the incorrect format.

[0041] Specifically, data standardization is to unify the data format so that all call logs have the same structure and format. For example, all request parameters and response parameters can be converted to JSON format. Timestamps can also be uniformly converted to a specific time format to facilitate time series analysis.

[0042] Optionally, determining the field characteristics of the log data includes: determining the data type and field value of each field in the log data; determining the minimum and maximum values ​​of each field based on the field value, and generating a value range for each field based on the minimum and maximum values; analyzing the frequency distribution of the field values ​​to determine the distribution characteristics of each field; and using the data type, value range and distribution characteristics as field characteristics.

[0043] The data type of each field can be determined by analyzing the log content. For example, if the field content is composed of numbers and no decimal part, it can be judged as an integer type; if the field content contains a decimal point, it may be a floating point type; if the field content is a text composed of letters, numbers and special characters, it may be a string type; if the field content represents the date and time format, it is a date and time type, etc.

[0044] Specifically, by traversing each record in the log data, the field values ​​of each field can be extracted. When determining the minimum and maximum values, if it is a numeric type, you can directly compare the size to find the minimum and maximum values; if it is a string type, you can compare it in lexicographic order; if it is a date and time type, you can convert it to a timestamp for comparison. For example, for a numeric field, you can use algorithms such as quick sort to sort the field values, and then take the first and last values ​​as the minimum and maximum values, respectively. After determining the minimum and maximum values, you can generate a value range. For a numeric field, the value range is the interval from the minimum value to the maximum value; for a string field, the value range can be a set of all possible strings; for a date and time field, the value range is the time period from the earliest time point to the latest time point.

[0045] Specifically, the distribution characteristics can be determined by counting the frequency of occurrence of different field values. For example, for a field representing the user status, there may be several states such as "online", "offline", and "busy", and the number of times each state appears in the log data is counted. Common distribution characteristics include uniform distribution, normal distribution, and skewed distribution. If the frequency of occurrence of field values ​​in each value range is roughly equal, then it is a uniform distribution; if the field values ​​show a bell-shaped curve distribution, then it is a normal distribution; if the field values ​​are concentrated on one side or in a few value ranges, then it is a skewed distribution.

[0046] For example, the data type of the field age of platform A is integer, the corresponding value range is 1-100, the corresponding distribution characteristic is normal distribution, and the average value is about 35. The value range of the field yearsOld of platform B is 1-100, the corresponding distribution characteristic is normal distribution, and the average value is about 34.

[0047] S120. Match the log data of the two platforms based on the field features to generate a total matching value, and generate a mapping relationship according to the total matching value.

[0048] Among them, the total match value refers to the degree of match between field pairs calculated by comparing and calculating multiple field features of the log data of the two platforms. The mapping relationship refers to the corresponding relationship established between the log data of the two platforms. After the total match value is generated by comparing the field features, it is possible to determine which fields on the two platforms correspond to each other, thereby establishing a mapping relationship.

[0049] Optionally, the log data of the platforms on both sides are matched based on field characteristics to generate a total matching value, including: matching the fields of the platforms on both sides based on field characteristics to determine the matching degree of each field pair, wherein the matching degree includes data type matching degree, value range matching degree and distribution characteristic matching degree; obtaining weight parameters corresponding to the characteristics of each field, and combining the weight parameters and the corresponding matching degree to generate a total matching value for each field pair.

[0050] Specifically, the matching degree includes data type matching degree, value range matching degree and distribution feature matching degree. Data type matching degree is directly obtained by checking the compatibility of data types through a predefined compatibility matrix. Value range matching degree quantifies the similarity of two fields in value range by calculating the overlap of field value ranges. Distribution feature matching degree uses statistical test methods to compare the data distribution patterns of two fields and obtain the degree of distribution similarity.

[0051] The purpose of data type matching is to evaluate the compatibility between the data types of two fields and ensure that the data can be converted and transmitted correctly. For example, integer types have a high degree of compatibility with integer types, while integer types have a low degree of compatibility with string types.

[0052] In a specific implementation, the controller may create a data type compatibility matrix, where each element in the matrix represents the degree of compatibility between two data types, with a value range of 0 to 1. The data types of the two fields are then used to find the corresponding data type matching degree in the compatibility matrix. A data type matching degree of 1.0 indicates full compatibility, and a lower value indicates poorer compatibility, which may require complex type conversion.

[0053] Among them, the value range matching can evaluate whether the value ranges of two fields are similar to determine whether they may represent the same meaning. If the value ranges of the corresponding fields on both platforms have a large overlap, it means that the data of the two fields are similar in value and the matching degree is high; otherwise, the matching degree is low.

[0054] In a specific implementation, the minimum and maximum values ​​of the corresponding fields of the platforms on both sides can be determined to determine the value range. Then calculate the proportion of the overlapping parts of the two value ranges to their respective value ranges. For example, the value range of field A is [1,10], and the value range of field B is [5,15]. Then the overlapping part is [5,10], accounting for 60% of the value range of field A and 50% of the value range of field B. The average of these two proportions can be taken as the value range matching degree, that is, 55%, which can be converted into a value range matching degree according to actual conditions, such as 0.55.

[0055] The distribution characteristics describe the distribution of data at different values. If the distribution characteristics of the corresponding fields on both platforms are similar, such as both are normally distributed with similar means and standard deviations, or both are skewed with similar skew directions and degrees, then the matching degree is high; otherwise, the matching degree is low.

[0056] In a specific embodiment, a statistical method can be used to compare the distribution characteristics. For example, for a normal distribution, the mean and standard deviation can be compared; for a skewed distribution, the skew coefficient can be compared, etc. Different matching values ​​are assigned according to the degree of difference. If the difference is very small, a higher distribution characteristic matching degree is assigned, such as 0.8; if the difference is large, a lower distribution characteristic matching degree is assigned, such as 0.3.

[0057] Among them, the total matching degree = w1×data type matching degree + w2×value range matching degree + w3×distribution feature matching degree, among which w1, w2, w3 are weight parameters.

[0058] In a specific implementation, the weight parameters can be initially set based on expert experience. The data type matching weight w1: Since the consistency of the data type is the basis of field matching, the initial setting is relatively high, such as w1=0.4. The value range matching weight w2: The value range can reflect the size and range of the field value, and the initial setting is w2=0.3. The distribution feature matching weight w3: The distribution feature provides the frequency and pattern information of the field value, and the initial setting is w3=0.3. The controller can then optimize the initial parameters based on the training data, including: data collection and annotation and weight optimization process. The controller will first collect historical matching data, that is, collect confirmed correct and incorrect field matching pairs from past synchronization tasks. Then the correct matches are annotated as positive samples, and the incorrect matches are annotated as negative samples. When optimizing the weights, the controller will first define the loss function, using the matching error rate as the loss function, with the goal of minimizing the error rate. Then use the optimization algorithm (grid search) to adjust the values ​​of w1, w2, and w3. Exemplarily, a grid search can be used, and the system traverses all possible combinations in the value range [0,1] of w1, w2, and w3 with a step size of 0.1 to ensure that w1+w2+w3=1. For each set of weight parameters, the matching degree is calculated and the matching accuracy is evaluated. Finally, the weight combination with the highest matching accuracy is selected as the final weight. For example, if the data type matching degree is 0.7, the value range matching degree is 0.6, and the distribution feature matching degree is 0.8, the corresponding weight parameters are 0.3, 0.4, and 0.3, respectively, then the total matching value = 0.7*0.3+0.6*0.4+0.8*0.3=0.69.

[0059] It should be noted that when calculating the total matching degree, the weight of each matching degree can be adjusted according to business needs. At the same time, in order to ensure data quality, outliers and missing values ​​can be processed to improve the accuracy of matching degree calculation. In addition to the above three matching degrees, factors such as field name similarity and business logic can also be combined.

[0060] Optionally, a mapping relationship is generated based on the total matching value, including: when the total matching value is greater than a preset threshold, the field pair is used as a target field pair; and a mapping relationship between the platform fields on both sides is generated based on the corresponding relationship of the target field pair.

[0061] Specifically, the setting of the preset threshold needs to comprehensively consider multiple factors, including business needs, data characteristics, and requirements for matching accuracy. The controller calculates the total matching value of each field pair on both platforms. If the total matching value is greater than the preset threshold, the field pair is marked as a target field pair. The target field pair represents a combination of fields with high similarity on both platforms. By determining the corresponding relationship of the target field pairs, a mapping relationship between the fields on both platforms can be established. The mapping relationship can be one-to-one, one-to-many, or many-to-one, depending on the actual situation of the data. For example, if a field on one platform corresponds to multiple fields on the other platform, it is a one-to-many mapping relationship; conversely, if multiple fields on one platform correspond to one field on the other platform, it is a many-to-one mapping relationship.

[0062] Furthermore, for each target field pair, the names and attributes of the platform fields on both sides can be recorded, and then a mapping table or data structure can be established to store the corresponding relationship between the platform fields on both sides. After the mapping relationship is generated, further verification and adjustment can be performed. For example, by sampling and checking some data records, the accuracy of the mapping relationship can be confirmed; or users can also dynamically adjust and optimize the mapping relationship according to new data and business needs.

[0063] S130: Synchronize log data of both platforms according to the mapping relationship.

[0064] Specifically, after the mapping relationship is established, the controller can synchronize the log data according to the mapping relationship. For newly generated log data, the controller can quickly find the corresponding log record according to the mapping relationship and synchronize it to the other platform. During the synchronization process, the consistency and integrity of the data need to be considered. If the data on one platform changes, the change needs to be synchronized to the other platform in a timely manner to ensure that the log data on both platforms remain consistent. Through the data synchronization process, the integration and unified management of log data on both platforms can be achieved, providing strong support for subsequent data analysis and business decision-making.

[0065] The technical solution of the embodiment of the present invention can more accurately match data by respectively obtaining the field characteristics of the log data of the platforms on both sides, including data type, value range and distribution characteristics, thereby greatly improving the accuracy of data synchronization and reducing data errors and inconsistencies. The matching and mapping method based on field characteristics does not rely on a specific platform or data format, can adapt to various types and structures of log data, and has strong versatility and scalability. Effective synchronization of log data on the platforms on both sides is achieved, making it convenient for users to share data between different platforms.

[0066] Embodiment 2

[0067] Figure 2 This is a flowchart of a cross-platform log data synchronization method provided in the second embodiment of the present invention. This embodiment adds a specific process of synchronizing log data on both platforms according to the mapping relationship based on the above-mentioned first embodiment. Among them, the specific content of steps S210-S220 is roughly the same as steps S110-S120 in the first embodiment, so they will not be repeated in this embodiment. Figure 2 As shown, the method includes:

[0068] S210: Obtain log data from both platforms respectively, and determine field characteristics of the log data, wherein the field characteristics include data type, value range, and distribution characteristics.

[0069] Optionally, log data is obtained from both platforms, including: collecting call logs from both platforms, wherein the call logs include request and response parameters, call timestamps, and call results; performing data cleansing on the call logs to generate cleaned call logs; and performing data standardization on the cleaned call logs to generate log data.

[0070] Optionally, determining the field characteristics of the log data includes: determining the data type and field value of each field in the log data; determining the minimum and maximum values ​​of each field based on the field value, and generating a value range for each field based on the minimum and maximum values; analyzing the frequency distribution of the field values ​​to determine the distribution characteristics of each field; and using the data type, value range and distribution characteristics as field characteristics.

[0071] S220. Match the log data of the two platforms based on the field features to generate a total matching value, and generate a mapping relationship according to the total matching value.

[0072] Optionally, the log data of the platforms on both sides are matched based on field characteristics to generate a total matching value, including: matching the fields of the platforms on both sides based on field characteristics to determine the matching degree of each field pair, wherein the matching degree includes data type matching degree, value range matching degree and distribution characteristic matching degree; obtaining weight parameters corresponding to the characteristics of each field, and combining the weight parameters and the corresponding matching degree to generate a total matching value for each field pair.

[0073] Optionally, a mapping relationship is generated based on the total matching value, including: when the total matching value is greater than a preset threshold, the field pair is used as a target field pair; and a mapping relationship between the platform fields on both sides is generated based on the corresponding relationship of the target field pair.

[0074] S230. Determine the field to be synchronized corresponding to the first platform according to the mapping relationship.

[0075] S240: Convert the fields to be synchronized based on the mapping relationship to generate conversion data.

[0076] S250: Store the converted data on the second platform.

[0077] The mapping relationship is generated by analyzing and matching the field features of the log data on both platforms, which clarifies the correspondence between the fields on the first platform and the second platform. For example, if the first platform has a field named "user_id", the mapping relationship can determine that the corresponding field name on the second platform is "customer_id", then when synchronizing data, "user_id" is one of the fields to be synchronized from the first platform to the second platform.

[0078] Specifically, the controller will traverse all fields of the first platform and find out whether each field has a corresponding field in the second platform based on the mapping relationship. If there is a corresponding field, then this field is marked as a field to be synchronized. Then, according to the mapping relationship, the field to be synchronized is further converted to generate conversion data, and the conversion data is stored in the second platform, realizing the log data synchronization of the two platforms (the first platform and the second platform), ensuring the data consistency and integrity of the two platforms, and providing a reliable data foundation for subsequent data analysis and business applications.

[0079] Optionally, the method also includes: monitoring the data synchronization process based on preset monitoring indicators; when an abnormal situation is monitored, determining the type of abnormality, and determining adjustment parameters based on the abnormality type, wherein the abnormality types include data type mismatch, field value out of range, field missing, and inconsistent nested structure; adjusting the mapping relationship based on the adjustment parameters.

[0080] The preset monitoring indicators are the measurement standards set to ensure the accuracy and stability of the data synchronization process. The preset monitoring indicators include the frequency, data volume, time consumption, success rate and error rate of data synchronization. For example, the data is synchronized once an hour, the data volume of each synchronization should be within a certain range, and the synchronization time should not exceed a specific duration.

[0081] Specifically, when the monitoring indicators exceed the normal range or other abnormal situations occur, it is necessary to determine the abnormality type so as to take targeted measures to deal with it. The abnormality types include data type mismatch, field value out of range, field missing, and inconsistent nested structure. Data type mismatch means that if the data type of a field is found to be inconsistent between the two platforms during the synchronization process, it is a data type mismatch exception. For example, a field on the first platform is an integer type, but when it is synchronized to the second platform, it is found that the corresponding field is a string type. Field value out of range means that when the value of a field exceeds the expected value range, a field value out of range exception occurs. For example, a field representing age has a value range of 0 to 120 in the first platform, but after synchronization to the second platform, it is found that there is a value greater than 120. Field missing means that if some fields that should exist are found to have no data in the second platform, it is a field missing exception. It may be that data loss or mapping errors occurred during the synchronization process, resulting in some fields not being synchronized correctly. Inconsistent nested structures means that if there is a nested structure in the data (such as nested fields in a JSON object or an associated table in a database), and the nested structures in the two platforms are inconsistent, a nested structure inconsistency exception will occur. For example, a JSON object in the first platform contains a nested array field, but the corresponding object in the second platform does not have this array field.

[0082] Furthermore, different types of exceptions require different adjustment parameters to fix the problem. The determination of the adjustment parameters needs to be combined with the characteristics of the exception type and the specific situation of data synchronization. For data type mismatch exceptions, the adjustment parameters can be determined as data type conversion rules. For example, if the field of the first platform is of integer type and the field of the second platform is of string type, the adjustment parameters can be determined as the rule for converting integers to strings. For field value out-of-range exceptions, the adjustment parameters can be determined as the verification rules and processing methods for the value range. For example, it can be set that when the value is found to be out of range, the value is replaced with the default value or an alarm is issued for manual processing. For field missing exceptions, the adjustment parameters can be determined as rechecking the mapping relationship and data synchronization process to ensure that all fields can be synchronized correctly. It may be necessary to adjust the mapping relationship or fix errors in the data synchronization process. For nested structure inconsistency exceptions, the adjustment parameters can be determined as rules for analyzing and adjusting the nested structure. For example, if the nested array field of the first platform has no corresponding field in the second platform, you can consider adding corresponding fields in the second platform or adjusting the data synchronization method to ensure the consistency of the nested structure.

[0083] It can be known that the mapping relationship is the basis of data synchronization. When an abnormal situation occurs, the controller needs to fix the problem in data synchronization by adjusting the mapping relationship to ensure the accuracy and completeness of the subsequent data synchronization process.

[0084] Furthermore, after automatically adjusting the mapping relationship, the controller will record these adjustments and use machine learning techniques to continuously optimize future mapping decisions, including: feedback-based temptation, dynamic optimization of mapping rules, and machine school optimization.

[0085] Among them, feedback-based optimization includes user feedback and historical data accumulation. User feedback means that after the controller automatically adjusts the mapping relationship, it will present the adjustment results for user confirmation. The user can choose to accept or modify these adjustments, and the controller adjusts future mapping decisions based on user feedback. Historical data accumulation means that the controller will save all exception repair records and mapping adjustment history as a reference for similar situations in the future. Dynamic optimization of mapping rules means that by monitoring the success rate and exception frequency of data synchronization, the controller can automatically adjust the mapping rules and give priority to the mapping method that performs well in historical data. Exemplarily, if the controller detects that a certain field often has exceptions many times, it will dynamically adjust the mapping rules to give priority to avoiding incorrect data type or structure mapping. Machine learning optimization includes abnormal pattern recognition and automatic weight adjustment. Abnormal pattern recognition means that the controller can analyze historical abnormal data through machine learning algorithms (such as clustering and classification algorithms), identify the pattern of abnormal occurrence, and prevent the same type of abnormalities in future data synchronization. Automatic weight adjustment means that in the field matching process, the controller adjusts the weight parameters in the matching algorithm through continuous learning, so that the field matching degree calculation is more accurate.

[0086] The technical solution of the embodiment of the present invention can more accurately match data by respectively obtaining the field characteristics of the log data of the platforms on both sides, including data type, value range and distribution characteristics, thereby greatly improving the accuracy of data synchronization and reducing data errors and inconsistencies. The matching and mapping method based on field characteristics does not rely on a specific platform or data format, can adapt to various types and structures of log data, and has strong versatility and scalability. Effective synchronization of log data on the platforms on both sides is achieved, making it convenient for users to share data between different platforms.

[0087] Embodiment 3

[0088] Figure 3 This is a schematic diagram of the structure of a cross-platform log data synchronization device provided by Embodiment 3 of the present invention. Figure 3 As shown, the device includes: a log data acquisition module 310, which is used to acquire log data from both platforms and determine the field characteristics of the log data, wherein the field characteristics include data type, value range and distribution characteristics;

[0089] The log data matching module 320 is used to match the log data of the two platforms based on the field features to generate a total matching value, and generate a mapping relationship according to the total matching value;

[0090] The log data synchronization module 330 is used to synchronize the log data of the two platforms according to the mapping relationship.

[0091] Optionally, the log data acquisition module 310 specifically includes: a log data acquisition unit, used to: collect call logs from both platforms, wherein the call logs include request and response parameters, call timestamps, and call results; perform data cleansing on the call logs to generate cleaned call logs; and perform data standardization on the cleaned call logs to generate log data.

[0092] Optionally, the log data acquisition module 310 specifically includes: a field feature determination unit, used to: determine the data type and field value of each field in the log data; determine the minimum and maximum values ​​of each field based on the field value, and generate a value range for each field based on the minimum and maximum values; analyze the frequency distribution of the field value to determine the distribution characteristics of each field; and use the data type, value range and distribution characteristics as field characteristics.

[0093] Optionally, the log data matching module 320 specifically includes: a total matching value determination unit, which is used to: match the fields of the two platforms based on the field characteristics to determine the matching degree of each field pair, wherein the matching degree includes the data type matching degree, the value range matching degree and the distribution characteristic matching degree; obtain the weight parameter corresponding to each field feature, and combine the weight parameter and the corresponding matching degree to generate the total matching value of each field pair.

[0094] Optionally, the log data matching module 320 specifically includes: a mapping relationship generating unit, which is used to: when the total matching value is greater than a preset threshold, use the field pair as a target field pair; and generate a mapping relationship between the platform fields on both sides according to the corresponding relationship of the target field pair.

[0095] Optionally, the log data synchronization module 330 is specifically used to: determine the to-be-synchronized fields corresponding to the first platform according to the mapping relationship; convert the to-be-synchronized fields based on the mapping relationship to generate conversion data; and store the conversion data to the second platform.

[0096] Optionally, the device also includes: a synchronization data monitoring module, which is used to: monitor the data synchronization process based on preset monitoring indicators; when an abnormal situation is monitored, determine the type of abnormality, and determine adjustment parameters based on the abnormality type, wherein the abnormality types include data type mismatch, field value out of range, field missing and inconsistent nested structure; adjust the mapping relationship based on the adjustment parameters.

[0097] The technical solution of the embodiment of the present invention can more accurately match data by respectively obtaining the field characteristics of the log data of the platforms on both sides, including data type, value range and distribution characteristics, thereby greatly improving the accuracy of data synchronization and reducing data errors and inconsistencies. The matching and mapping method based on field characteristics does not rely on a specific platform or data format, can adapt to various types and structures of log data, and has strong versatility and scalability. Effective synchronization of log data on the platforms on both sides is achieved, making it convenient for users to share data between different platforms.

[0098] A cross-platform log data synchronization device provided by an embodiment of the present invention can execute a cross-platform log data synchronization method provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0099] Embodiment 4

[0100] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0101] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0102] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0103] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as a cross-platform log data synchronization method.

[0104] In some embodiments, a cross-platform log data synchronization method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the cross-platform log data synchronization method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform a cross-platform log data synchronization method in any other appropriate manner (e.g., by means of firmware).

[0105] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0106] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0107] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0108] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0109] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0110] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0111] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0112] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A cross-platform log data synchronization method, characterized in that: include: Obtain log data from both platforms respectively, and determine field characteristics of the log data, wherein the field characteristics include data type, value range, and distribution characteristics; Matching the log data of the two platforms based on the field features to generate a total matching value, and generating a mapping relationship according to the total matching value; The log data of both platforms are synchronized according to the mapping relationship.

2. The method according to claim 1, characterized in that The log data is obtained from the two platforms respectively, including: Collect call logs from both platforms, where the call logs include request and response parameters, call timestamps, and call results; Performing data cleaning on the call log to generate a cleaned call log; The cleaned call log is subjected to data standardization processing to generate the log data.

3. The method according to claim 1, characterized in that The determining of the field characteristics of the log data includes: Determine the data type and field value of each field in the log data; Determine the minimum value and the maximum value of each field according to the field value, and generate the value range of each field according to the minimum value and the maximum value; Analyzing the frequency distribution of the field values ​​to determine the distribution characteristics of each field; The data type, the value range and the distribution characteristics are used as the field characteristics.

4. The method according to claim 3, characterized in that The matching of the log data of the two platforms based on the field features to generate a total matching value includes: Matching the fields of the two platforms based on the field characteristics to determine the matching degree of each field pair, wherein the matching degree includes data type matching degree, value range matching degree and distribution feature matching degree; The weight parameter corresponding to each field feature is obtained, and the weight parameter and the corresponding matching degree are combined to generate a total matching value for each field pair.

5. The method according to claim 4, characterized in that Generating a mapping relationship according to the total matching value includes: When the total matching value is greater than a preset threshold, the field pair is used as a target field pair; A mapping relationship between the platform fields on both sides is generated according to the corresponding relationship of the target field pair.

6. The method according to claim 5, characterized in that The two-side platforms include a first platform and a second platform, and synchronizing the log data of the two-side platforms according to the mapping relationship includes: Determine, according to the mapping relationship, a field to be synchronized corresponding to the first platform; Converting the to-be-synchronized field based on the mapping relationship to generate conversion data; The converted data is stored in the second platform.

7. The method according to claim 1, characterized in that The method further comprises: Monitor the data synchronization process based on preset monitoring indicators; When an abnormal situation is monitored, the abnormal type is determined, and adjustment parameters are determined according to the abnormal type, wherein the abnormal type includes data type mismatch, field value out of range, field missing and nested structure inconsistency; The mapping relationship is adjusted based on the adjustment parameter.

8. A cross-platform log data synchronization device, characterized in that: include: A log data acquisition module, used to acquire log data from both platforms and determine field characteristics of the log data, wherein the field characteristics include data type, value range and distribution characteristics; A log data matching module, used to match the log data of the two platforms based on the field features to generate a total matching value, and generate a mapping relationship according to the total matching value; The log data synchronization module is used to synchronize the log data of the two platforms according to the mapping relationship.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The computer storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method according to any one of claims 1 to 7 when executed.

Images