Procurement platform heterogeneous data fusion and visualization processing method

Abstract

The invention relates to a heterogeneous data fusion and visualization processing method for a purchase platform, and belongs to the technical field of heterogeneous data fusion, and the method specifically comprises the steps: firstly, carrying out the deep cleaning of data according to a time sequence, precisely removing redundant information, guaranteeing the consistency and timeliness of the data to be effectively maintained, and solving the problems of data redundancy and timeliness, a mode of flexibly setting a weight ratio according to different time points and data sources is adopted, so that the single limitation of a traditional data processing mode is broken through, meanwhile, fine matching and integration processing are performed on data from different time points and different channels, data resources of building enterprises are greatly enriched, and the data processing efficiency is improved. And more accurate and comprehensive data insight ability is brought to the system.

Description

Technical Field

[0001] The invention relates to a procurement platform heterogeneous data fusion and visualization processing method, belonging to the technical field of heterogeneous data fusion. Background Art

[0002] With the development of computer technologies such as big data and cloud computing, the production data of construction enterprises has also become an important part of enterprise assets. Enterprise asset data includes data sources from systems such as personnel, procurement, finance and taxation. The data may be in the form of pictures, numbers, interfaces and other structures, which cannot be directly used to generate value.

[0003] At present, construction companies are facing the dilemma of poor use of system data. The data correlation between their various information systems is low or almost unrelated, which seriously restricts the improvement of the information level of construction companies. In the process of data cleaning, redundant data cannot be effectively identified and processed, resulting in data redundancy and inconsistency in the process of data fusion. When deduplicating data, the integrity of the data cannot be fully ensured, which may cause information loss or a decrease in data accuracy. At the same time, in the process of data fusion, due to the lack of the ability to dynamically adjust data weights according to the situation, and the inability to fully guarantee data integrity, the timeliness of the data is greatly reduced. More importantly, the current data fusion mainly relies on a single data source, ignoring other critical information elements, which greatly limits the value mining potential of the data. Summary of the invention

[0004] In order to solve the technical problems existing in the prior art, the present invention provides a method for heterogeneous data fusion and visualization processing of a procurement platform, which can significantly improve the utilization efficiency of data by implementing multi-source data fusion and weight matching strategy.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is a method for heterogeneous data fusion and visualization processing of a procurement platform, which is operated according to the following steps: a. Data collection: collecting various procurement microservice databases, enterprise finance and taxation databases, enterprise query databases, and various operation records and procurement information data of buyers and suppliers on the platform; b. Data cleaning: In the data from the same source, the old redundant data is completely removed according to the data with the new timestamp. For the data of the same field that is different, the weight of the field in the source data is compared, and the data is adjusted according to the weight to ensure uniformity. For abnormal data, the abnormal data is supplemented according to the previous timestamp to obtain the correction value of the abnormal data point; c. Data fusion: The processed real-time data is integrated with the actual service. At the same time, the fused data is segmented and counted by keywords. Fields with keywords exceeding the threshold are labeled with user identities. Business opportunities are matched with user tags, and business opportunities are pushed to users in real time. d. Set the data decay factor for the user label obtained by the fusion data, update the label, and calculate the time decay factor based on the timeliness and change rate of the data; e. Data visualization: the integrated data is displayed in the indicators of the data screen in the form of reports and charts.

[0006] Preferably, in step b, after data cleaning, the cleaned data is reprocessed, and a preliminary evaluation of the data is performed by examining the accuracy of historical data, the update frequency of the data source, and the length of time the data source exists, while introducing the error rate of the data source and the data integrity as key evaluation factors; For the timestamp data, the accuracy of the data is checked and adjusted by the difference of different time series. The absolute difference between the corresponding data points at two different time points is taken, and the absolute difference is compared with the two boundary points and the middle value. Perform a dot product operation on the vectors of each pair of data items, that is, calculate the sum of the products of their corresponding elements, then calculate the modulus of the two vectors, that is, the square root of the sum of the squares of their respective elements, and finally divide the result of the dot product by the product of the two moduli. The resulting ratio is the cosine similarity between the two data items; compare the calculated similarity with the threshold, and re-correct the data that exceeds the threshold range; In the data alignment mechanism, weights are used to perform weighted averaging on the data to ensure data consistency.

[0007] Preferably, in step d, a time decay factor is calculated, and the time decay factor decayi is defined as: d , where Tcurrent is the current timestamp, i is the different data sources, and T is the timestamp parameter. Combining the data source weight p1, the time decay factor n and the attenuation parameter o, the field weight Fij can be defined as: Fij=p1i*di*(1−oi)*Xij, where X is the field value, The comprehensive weight Fjtotal on all data sources is summarized and further calculated: Fjtotal= Fij The process of calculating weights can be optimized through conditional judgment: when a certain condition (for example, F is less than 0) is met, the corresponding method will be triggered to remove specific keywords from the label. Then, the i data are quickly sorted, and the data with the largest weight after sorting is determined as the confirmation value of the same field.

[0008] Compared with the prior art, the present invention has the following technical effects: The present invention is committed to integrating the diversified service data on the procurement platform and its business-related system data, and significantly improves the data quality and timeliness through a series of refined processing processes. First, the collected data is deeply cleaned to effectively eliminate redundant information and ensure the accuracy and purity of the data. On this basis, an event timestamp is added to each data item, and a dynamic time planning algorithm is used to accurately measure the similarity between different time series, thereby realizing instant updates of data and effective recovery of abnormal nodes, further enhancing the timeliness and integrity of the data.

[0009] At the same time, in view of the common problems in the field of data fusion in the construction industry, such as single fusion mode, excessive redundant data, lack of data consistency, neglect of time series, and rigid weight mechanism. By deeply integrating the data resources of various microservices of recruitment and procurement, it is possible to accurately match and push corresponding business opportunity information based on user feature data sets. On this basis, the secondary fusion of fused data and other official system data is also realized. With the help of these more authoritative data sources, the fused data is corrected and supplemented, so as to establish a complete user rating system or other customized applications. This innovative data fusion strategy not only meets the company's needs for deep mining and secondary utilization of business data, but also makes full use of the high correlation between the company's relevant systems, providing strong support for the company's data-driven decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a data flow chart for the procurement of multi-source heterogeneous data fusion in Example 1 of the present invention. DETAILED DESCRIPTION

[0011] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Embodiment 1

[0012] like Figure 1 As shown in the figure, a procurement platform heterogeneous data fusion and visualization processing method is performed according to the following steps: Step 1: Collect data. For a microservice architecture system, each platform will contain many microservices, and the way to store data is no longer just a database structure. At the same time, collect interface return values ​​and API services, etc. There are two ways to collect databases.

[0013] The first method is to use the interface call method to accurately extract the required data set from the system database. Then, these key data are transferred to an intermediate database for subsequent use. The significant advantage of this approach is that it can ensure that the data in the intermediate database is synchronized with the official database in real time or regularly, thereby improving the timeliness and accuracy of the data. However, it also has certain limitations, that is, it may cause additional load on the operation of the official database, increasing the pressure on system processing.

[0014] The second method is to restore the data backup to obtain data. Its significant advantage is that this operation will not cause any interference or impact on the operation of the formal database, thus ensuring the stability and continuity of the formal database. However, this method also has certain shortcomings, that is, there may be a certain time difference between the data backup and the formal database, which means that the restored data may not be the latest and cannot fully reflect the latest status of the formal database after the backup. Step 2: Clean and merge the data. First, merge the multi-source table structure of tables and queues, label the multiple fields of the data source with weights, segment the words by keywords, add a timestamp to each data source through the interface, convert the timestamp into a digital type and add it to the weight calculation.

[0015] For tasks that process text and images, we first use convolutional neural networks (CNNs) to extract features from images, and recurrent neural networks (RNNs) to capture temporal and semantic features in text. Then, we organically fuse the features from these two different sources in the middle layer. Data with different temporal and spatial distributions need to be preprocessed and aligned.

[0016] Two sets of timestamps are used, one from the video frames and the other from the audio frames. The timestamps of the video frames are from 0 to 10 seconds, and there are 25 frames per second, so the time interval is 0.04 seconds. The timestamps of the audio frames are also from 0 to 10 seconds, but there are 100 frames per second, so the time interval is 0.01 seconds. This means that the sampling rate of the audio frames is higher than the sampling rate of the video frames.

[0017] To align the video and audio data, you need to find the nearest audio frame corresponding to each video frame timestamp. By using the sorting function, find the index of the insertion point in the sorted array to maintain the sort order of the array. By finding the position in the audio array where each video value should be inserted, you can get the index of the audio frame closest to the video frame timestamp. Through data cleaning, you can get the user's feature data set.

[0018] Step 3: Extract and count keywords through the word segmenter, set the keyword threshold to x, and if the keyword statistics value is greater than the threshold, a label will be formed and added to the user's data. At the same time, set the attenuation parameter, which can be calculated and designed according to the number of operations or time.

[0019] Step 4: Match user tags with business opportunity attributes to form a publish-subscribe model, so that customers can obtain more possible business opportunity information and accelerate the transaction volume of the platform.

[0020] When calculating the change weight ratio, it is necessary to carefully distinguish different data sources, where i represents different data sources. Each data source has its specific field weight ratio p1, the field involved is X, the timestamp parameter is recorded as T, and the proportion of time is p2. In addition, the attenuation parameter o and the attenuation factor n are also considered, which together affect the calculation of the field weight F.

[0021] Specifically, the field weight F of each data source is determined based on its field value X, field weight ratio p1, timestamp T (the time decay effect calculated by time ratio p2 and decay factor n), and decay parameter o. This allows for flexible adjustment of the importance of different data sources and fields in the overall evaluation, while also taking into account the freshness of time and the decay of the data.

[0022] Calculate the time decay factor. The time decay factor di can be defined as: d , where Tcurrent is the current timestamp.

[0023] Combining the data source weight, time decay factor and attenuation parameter, the field weight Fij can be defined as: Fij=p1i*di*(1−oi)*Xij The comprehensive weights of all data sources are summarized and further calculated: Fjtotal= Fij The process of calculating weights can be optimized through conditional judgment: when a certain condition (for example, F is less than 0) is met, the corresponding method will be triggered to remove specific keywords from the label. Then, the i data are quickly sorted, and the data with the largest weight after sorting is determined as the confirmation value of the same field.

[0024] Step 6: In addition to matching and distributing business opportunities, early warning and evaluation mechanisms can also be implemented for user behavior. The fused data is compared and supplemented with data from other official systems, and similar data is extracted from financial and corporate information bases. The corresponding calculation formulas are applied to comprehensively evaluate their weights to obtain their respective data results. Subsequently, these results are matched with the intermediate table, and the corresponding scores are calculated based on the matching results.

[0025] A multi-source heterogeneous data fusion algorithm is used to integrate the cleaned data, and then visualization tools are used to intuitively present the fused data, aiming to improve data quality and accelerate the process of data fusion visualization while ensuring its accuracy.

[0026] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the scope of the present invention.

Claims

1. A procurement platform heterogeneous data fusion and visualization processing method, characterized by: Follow the steps below, a. Data collection: collecting various procurement microservice databases, enterprise finance and taxation databases, enterprise query databases, and various operation records and procurement information data of buyers and suppliers on the platform; b. Data cleaning: In the data from the same source, the old redundant data is completely removed according to the data with the new timestamp. For the data of the same field that is different, the weight of the field in the source data is compared, and the data is adjusted according to the weight to ensure uniformity. For abnormal data, the abnormal data is supplemented according to the previous timestamp to obtain the correction value of the abnormal data point; c. Data fusion: The processed real-time data is integrated with the actual service. At the same time, the fused data is segmented and counted by keywords. Fields with keywords exceeding the threshold are labeled with user identities. Business opportunities are matched with user tags, and business opportunities are pushed to users in real time. d. Set the data decay factor for the user label obtained by the fusion data, update the label, and calculate the time decay factor based on the timeliness and change rate of the data; e. Data visualization: the integrated data is displayed in the indicators of the data screen in the form of reports and charts.

2. According to claim 1, a procurement platform heterogeneous data fusion and visualization processing method is characterized by: In step b, after data cleaning, the cleaned data is reprocessed, and a preliminary evaluation of the data is performed by examining the accuracy of historical data, the update frequency of the data source, and the length of time the data source exists, while introducing the error rate of the data source and the data integrity as key evaluation factors; For the timestamp data, the accuracy of the data is checked and adjusted by the difference of different time series. The absolute difference between the corresponding data points at two different time points is taken, and the absolute difference is compared with the two boundary points and the middle value. Perform a dot product operation on the vectors of each pair of data items, that is, calculate the sum of the products of their corresponding elements, then calculate the modulus of the two vectors, that is, the square root of the sum of the squares of their respective elements, and finally divide the result of the dot product by the product of the two moduli. The resulting ratio is the cosine similarity between the two data items; compare the calculated similarity with the threshold, and re-correct the data that exceeds the threshold range; In the data alignment mechanism, weights are used to perform weighted averaging on the data to ensure data consistency.

3. The method for heterogeneous data fusion and visualization processing of a procurement platform according to claim 1, characterized in that: In the step d, the time decay factor is calculated, and the time decay factor di is defined as: d , where Tcurrent is the current timestamp, i is the different data sources, and T is the timestamp parameter. Combining the data source weight p1, the time decay factor n and the attenuation parameter o, the field weight Fij can be defined as: Fij=p1i*di*(1−oi)*Xij, where X is the field value, The comprehensive weight Fjtotal on all data sources is summarized and further calculated: Fjtotal= Fij The process of calculating weights can be optimized through conditional judgment: when a certain condition (for example, F is less than 0) is met, the corresponding method will be triggered to remove specific keywords from the label. Then, the i data are quickly sorted, and the data with the largest weight after sorting is determined as the confirmation value of the same field.

Images