Cross-network program data updating method and system

Abstract

The invention provides a cross-network program data updating method and system, and relates to the technical field of cross-network programs, and the method comprises the steps: obtaining to-be-updated data from a data source in at least one first network environment, and carrying out the preprocessing of the to-be-updated data, so as to generate a data package meeting a cross-network transmission standard. Data formats in different network environments can be automatically converted, it is ensured that the data can be correctly analyzed and processed in the transmission process, an efficient data transmission mechanism is adopted, an optimized cross-network communication mode is adopted, the data transmission speed is increased, bandwidth occupation is reduced, it is ensured that the data can be rapidly and accurately transmitted to the target network environment, and the data transmission efficiency is improved. In the data transmission and storage process, security mechanisms such as encryption, authentication and access control are adopted, the security of data is ensured, data leakage, tampering and hostile attacks are prevented, an incremental updating algorithm is adopted, only changed data is transmitted and updated, the consumption of network bandwidth and computing resources is reduced, and the data updating efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cross-network program, in particular to a cross-network program data updating method and system. BACKGROUND

[0002] In today's digital age, cross-network program data updating has become an indispensable part of many fields, such as enterprise internal business systems, cross-regional distributed systems, and security systems involving multiple network environments. With the continuous development of network technology, the demand for data interaction between different network environments is increasing, and how to efficiently, securely and accurately implement cross-network program data updating has become a technical problem to be solved.

[0003] Limitations of traditional cross-network data updating methods: Different network environments often use different data formats and standards, and direct data transmission can result in incorrect parsing and processing of data, affecting the accuracy and effectiveness of data updating. Traditional data transmission methods may have slow transmission speed and high bandwidth occupation, especially when a large amount of data needs to be transmitted, which can result in a long data updating process and affect the normal operation of the system. During cross-network data transmission, data faces many security threats such as data leakage, tampering and malicious attacks. Lack of effective security mechanisms can compromise the security of data and pose potential risks to the system. Traditional data updating methods usually use full update, which requires retransmission and updating of all data each time. This method not only wastes a lot of network bandwidth and computing resources, but also is inefficient, especially when the data volume is large and the updating frequency is high. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a cross-network program data updating method and system that can improve the accuracy and comprehensiveness of data preprocessing for deformation monitoring.

[0005] To solve the above technical problems, the technical solution of the present application is as follows: In a first aspect, a cross-network program data updating method is provided, the method comprising: obtaining update data from at least one data source in a first network environment; preprocessing the update data to generate data packets conforming to cross-network transmission standards; obtaining update data from a first network environment; transforming the update data into a format supported by a second network environment; performing integrity verification and encryption processing on the transformed data to obtain encrypted data; Decrypt the encrypted data and calculate the difference between the data to be updated and the last transmission data using the difference algorithm to obtain incremental data; Add a mark to the incremental data, and transmit the marked incremental data from the first network environment to the second network environment through cross-network communication; Update the preprocessed data in the second network environment to obtain updated data; Return the updated data to the first network environment through cross-network communication; Apply the updated data in the first network environment.

[0006] Further, obtain the data to be updated from at least one data source in the first network environment, including: Determine the range of data to be updated from the database; Narrow the data source of the data to be updated; Determine that the data source is suitable for the first network environment; Establish a network connection between the data source and the first network environment; The data source is transmitted to the first network environment through the network connection to obtain the data to be updated.

[0007] Further, preprocess the data to be updated to generate data packets that meet the cross-network transmission standard, including: Collect the data to be updated, and determine that the collected data has a unified format; Clean the collected data to obtain accurate data; Standardize and format the accurate data to obtain converted data; Fuse the converted data to obtain a unified data structure; Predict the unified data to generate new data features; Statistical analysis of new data features, selecting relevant features of data, removing redundant features and reducing dimensions of data; Divide the data into training set and test set to obtain new data packets; Further, transmit the data packets from the first network environment to the second network environment through the preset cross-network communication method, including: Prepare and package the data packets to obtain packaged data packets; Verify the packaged data packets to ensure the accuracy of the packaged data transmission process to obtain accurate data; Transmit the accurate data through cross-network communication; Evaluate the cross-network method of transmitting from the first network environment to the second network environment; Evaluate the reliability and integrity of the cross-network communication method to obtain accurate communication methods; Ensure that the network device configuration of the communication method is correct; Transmit the data packet from the first network environment to the second network environment through the cross-network communication method.

[0008] Further, an optimization algorithm is used to update the data, and the optimization algorithm uses an incremental update algorithm, including: Pull the transmission data supporting incremental collection from the communication method; According to the collected transmission data, select an update model for calculation; Adjust the model parameters by calculating the increment; Calculate the transmission data by incremental learning to increase the gradient required for data update; Update the model parameters according to the calculated increment; According to the calculated increment, update the model parameters and integrate them into the newly added data; Optimize and adjust the newly added data during the incremental update process.

[0009] Further, the preprocessed data in the second network environment is processed and updated to obtain updated data, including: Preprocess the data in the second network environment to obtain preprocessed data; Format verification is performed on the preprocessed data to verify whether the data type conforms to the data model to obtain complete data; Compare the complete data with the target data in the second network environment to determine the consistency of the data; Update the complete data to obtain updated metadata; Perform sentiment analysis on the updated metadata to generate sentiment labels; Extract keywords in the sentiment labels using NLP technology and store them to obtain metadata fields; Update the metadata fields to obtain updated data.

[0010] Further, the updated data is returned to the first network environment through the cross-network communication method, and the updated data is applied in the first network environment, including: Determine that the cross-network communication method is safe and reliable, and configure a firewall for the cross-network communication method; Configure a secret key for the cross-network communication method, and desensitize the information to obtain secure data; Encapsulate the secure data to obtain encapsulated data; Securely transmit the encapsulated data from the second network environment to the first network environment to obtain transmission data; The transmitted data is received in the first network environment to obtain received data; The received data is verified to verify the integrity of the data to obtain complete data; The complete data is format-verified to verify the correctness of the format; The complete data is merged with the update data into the first network environment and stored.

[0011] In a second aspect, a cross-network program data update system is applied to the method and includes: An acquisition module acquires the data to be updated from a data source in at least one first network environment; A processing module pre-processes the data to be updated to generate a data packet conforming to a cross-network transmission standard; A calculation module transmits the data packet from the first network environment to a second network environment through a preset cross-network communication mode, and uses an optimization algorithm to update the data, the optimization algorithm uses an incremental update algorithm, and the specific steps are as follows: The data to be updated is acquired from the first network environment; The data to be updated is converted into a format supported by the second network environment; The converted data is integrity-verified and encrypted to obtain encrypted data; The encrypted data is decrypted, and a difference between the data to be updated and last transmission data is calculated using a difference algorithm to obtain incremental data; The incremental data is marked, and the marked incremental data is transmitted from the first network environment to the second network environment through the cross-network communication mode; An update module processes and updates the pre-processed data in the second network environment to obtain updated data; A return transmission module returns the updated data to the first network environment through the cross-network communication mode; An application module applies the updated data in the first network environment.

[0012] In a third aspect, a computing device includes: One or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of claim 8.

[0013] In a fourth aspect, a computer-readable storage medium stores a program, and the program is executed by a processor to implement the method.

[0014] The above-mentioned scheme of the present application at least includes the following beneficial effects.

[0015] 1. The method can automatically convert the data format in different network environments, ensure that the data can be correctly parsed and processed during transmission, and has a high-efficiency data transmission mechanism: an optimized cross-network communication method is adopted to improve the data transmission speed, reduce the bandwidth occupation, and ensure that the data can be quickly and accurately transmitted to the target network environment.

[0016] 2. In the process of data transmission and storage, encryption, authentication, access control and other security mechanisms are adopted to ensure the security of data, prevent data leakage, tampering and malicious attacks, and an incremental update algorithm is adopted to only transmit and update the changed data, reduce the consumption of network bandwidth and computing resources, and improve the data update efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a flowchart of a cross-network program data updating method provided by an embodiment of the present application.

[0018] Figure 2 is a schematic diagram of a cross-network program data updating system provided by an embodiment of the present application. DETAILED DESCRIPTION

[0019] The exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be accurately conveyed to those skilled in the art.

[0020] As shown in Figure 1 , an embodiment of the present application proposes a cross-network program data updating method, which comprises the following steps: obtaining the data to be updated from at least one data source in a first network environment; preprocessing the data to be updated to generate data packets conforming to cross-network transmission standards; obtaining the data to be updated from the first network environment; transforming the data to be updated into a format supported by the second network environment; performing integrity check and encryption processing on the transformed data to obtain encrypted data; decrypting the encrypted data and calculating the difference between the data to be updated and the last transmission data using a difference algorithm to obtain incremental data; adding a mark to the incremental data, and transmitting the marked incremental data from the first network environment to the second network environment through a cross-network communication method; processing and updating the preprocessed data in the second network environment to obtain updated data; The updated data is returned to the first network environment through cross-network communication; The updated data is applied in the first network environment.

[0021] In the embodiment of the present application, only the changed data is transmitted through the incremental update algorithm, the amount of data transmitted across the network is reduced, the bandwidth is saved, and the efficiency of data updating is improved, the data encryption and decryption processing ensures the security of the data in the transmission process, avoids data leakage or tampering, ensures the integrity of the data, synchronously updates the data in the two network environments, ensures the data consistency of the first network environment and the second network environment, reduces the risk caused by data inconsistency, the format conversion ensures that the data between the two network environments can be smoothly transmitted and processed, enhances the compatibility of the system, through the optimized incremental update mechanism, reduces the invalid data transmission, saves the system resources, and reduces the maintenance and operation cost.

[0022] In a preferred embodiment of the present application, the data to be updated is obtained from at least one data source in the first network environment, comprising: The range of data to be updated is determined from the database, specifically including: determining the range of data to be updated, filtering the range of data to be updated, and determining the update time range of the data; selecting a query tool from the database type, querying and filtering the data to be updated from the database; using the database tool to execute the designed query, verifying the query result through automatic operation, and ensuring that the data filtering condition is accurate; obtaining the data to be updated according to the query result, if the data amount is large, the data can be obtained in batches or in batches; using the update operation to update the filtered data; The range of data to be updated is narrowed down, specifically including: determining whether there is a new time range, a change in a specific state field, and an adjustment of the data priority level, determining which data needs to be updated, setting specific filtering conditions, and gradually narrowing the range of data to be filtered; first, the time range is limited to avoid processing outdated data and ensure that only data within the required time range is selected; second, the state field is used to represent the data state to ensure that only the data to be updated is obtained; the data is filtered according to the priority level, the data with high priority level is processed first, and then the data range is narrowed; if the data amount is large, the paging query or batch processing mode can be used to reduce the performance problems caused by one-time query; the database is optimized, an index can be added to the state field, and data aggregation can be performed on part of the query to reduce unnecessary data transmission and calculation; Determine that the data source is suitable for the first network environment, specifically including: understanding the network topology of the first network environment, confirming whether the data source is located inside or outside the network, evaluating whether the bandwidth and delay meet the requirements according to the size and range frequency of the data source, analyzing the security policy in the network environment; Confirming that the data source is structured data, unstructured data or stream data, determining that the data source supports compatibility with the first network environment, confirming the data format supported by the data source; Evaluate the network environment, evaluate the network access path, determine the stability of the data source, whether there is a high-availability configuration, determine whether there is a data backup strategy, and monitor the performance of the data source; Determine whether the user in the first network environment has the right to access the data source, test the connectivity of the data source, and perform small-scale data access tests, optimize the access path of the data source in the first network environment according to the test results, and monitor the access of the data source in the first network environment in real time; Establish a network connection between the data source and the first network environment, specifically including: understanding the use scenario of the data source, evaluating whether a higher bandwidth, low-delay connection is needed, and determining the access protocol supported by the data source; If the data source and the first network environment are both in the intranet, determine that they can communicate directly through the internal switch; If the data source is located in the external network, a VPN connection can be used to securely access the data source; If higher bandwidth and lower delay are needed, a dedicated line connection can be selected; If direct access to the data source is not possible, a proxy server can be configured to access the data source; Configure security rules on the router between the first network environment and the data source, allow access to specific ports, ensure that only authorized users can establish a connection and access data by forcing the use of an authentication mechanism, determine that the domain name resolution of the data source is correct, if the data source is located in different subnets or external networks, appropriate routing rules need to be configured, and determine that the data can be correctly transmitted in the network; The data source is transmitted to the first network environment through a network connection to obtain to-be-updated data, specifically including: determining the type of the data source and the content of the to-be-updated data, selecting a network connection mode according to the architecture of the data source and the first network environment, ensuring that the data source can be accessed by the first network environment, verifying that the firewall, router and port settings are correct; according to the access protocol of the data source, writing a request code to obtain the to-be-updated data, selecting a data transmission mode, filtering, filtering and converting the data when obtaining the data; checking the data integrity and accuracy transmitted by the data source, cleaning the data, removing invalid data, filling in missing values and processing duplicates; when transmitting data, an encryption protocol is used to ensure the secure transmission of data, and in the process of ensuring data access, only authorized systems can obtain data; if the data is incremental update, the difference between new data and old data needs to be determined according to the timestamp or identifier, if it is full update, the old data in the first network environment is directly replaced, if there is a time delay or asynchronous processing between the data source and the first network environment, a timing task or message queue can be designed to ensure real-time or periodic synchronization of data.

[0023] In the embodiment of the application, by explicitly defining the data update range and narrowing the data source, the consumption of computing and storage resources is reduced, the compatibility of the data source and the network environment is determined, and the efficient transmission of data is ensured. Establishing a stable network connection reduces the risk of data transmission failure caused by network problems, and by accurately obtaining to-be-updated data, the accuracy of the data in the system is ensured, which is helpful for subsequent update operations and system maintenance. Encryption and verification measures in the data transmission process can ensure the security of the data, avoid data leakage and tampering, and the smooth execution and efficient operation of the entire process help to improve the overall stability of the system and ensure that data update does not affect other system functions.

[0024] In a preferred embodiment of the application, the to-be-updated data is preprocessed to generate a data packet that meets the cross-network transmission standard, including: The to-be-updated data is collected, and the collected data is determined to have a unified format, specifically including: determining the type of the collected data, determining the source of the data, in order to ensure that all data can be uniformly processed, defining a unified format, unifying the names of the data, and ensuring that the field types of all data are consistent; processing the missing values of the data, removing invalid data or abnormal values, and ensuring that all data meet the business logic; unifying the structure of the data, verifying the data type of each field, and ensuring that the format and type of all data are unified; Data cleaning is performed on the collected data to obtain accurate data, specifically including: checking the data type of each field, determining which fields are necessary, which fields can be empty, and whether there are inconsistent cases, determining whether there are missing values, abnormal values and error values in the data; the missing values are deleted and filled, and the selected filling methods for missing values include: using mean, median and mode to fill, removing duplicate data and abnormal value processing; manual processing can be performed on the data, manually setting the rules for filtering abnormal values, performing format conversion and type unification processing on the data, ensuring that the data type of all fields is correct, and for cases requiring numerical analysis or linear regression, data standardization or normalization is necessary, to obtain accurate data: Standardization processing and format conversion are performed on the accurate data to obtain converted data, specifically including: basic cleaning and preprocessing of accurate data, checking for null values in the data, which can be filled, deleted or replaced with specific values according to the situation, ensuring data format consistency, unifying timestamp fields in the data, and removing spaces and special fields; numerical field standardization and classification field standardization processing are performed on the data, and date and time fields in the data are unified; the standardized and formatted data is verified to ensure there are no errors and the data format is consistent, and after the standardization processing is complete, the data is saved in a new format to obtain converted data; The converted data is fused to obtain a unified data structure, specifically including: determining different data sources to ensure consistency of structure, fields and types of each data source, and determining the data structure after fusion; during data fusion, unnecessary fields need to be removed, and duplicate data needs to be removed, and data fusion can be merged in different ways according to requirements, including: merging data by row, merging data by column and merging data by condition; inconsistent data needs to be processed during fusion, missing values need to be filled and data consistency needs to be checked, redundant duplicate data needs to be removed, for time series data, the fused data needs to be filled with missing timestamps to ensure data consistency, and the unified data is verified to ensure data content consistency; The unified data is predicted to generate new data features, specifically including: collecting and unifying the data, after the complete collection of the data, pre-processing the data to remove irrelevant information, segmenting the text data, removing meaningless common words, removing special symbols in the text, and converting the text data into digital features that can be processed by a linear regression model; feature extraction is performed on the data, including sentiment analysis, keyword extraction, topic modeling, user behavior features, and time features, the extracted features are used for prediction tasks, and the specific steps include data planning, model selection, model training, and model evaluation; new data is predicted using the model to generate new prediction features; Statistical analysis is performed on the new data features, relevant features of the data are selected, redundant features are removed and the dimensionality is reduced, specifically including: distribution analysis is performed on the newly generated features, descriptive statistics are used to retrieve basic statistics of the data, the data is visualized, missing values are checked for each feature, and outliers are detected; correlation analysis and feature importance scoring are performed on the data, and the importance of each feature is evaluated using a random forest model; the data is correlated to remove redundancy, and redundant features are removed based on the model; the dimensionality of the data is reduced to improve computational efficiency and avoid overfitting; The data is divided into training and test sets to obtain new data packets, specifically including: before starting to divide the data set, preparing the data and importing it into the necessary repository; the data is randomly divided into training and test sets according to the given ratio; the size of the training and test sets can be verified after division to ensure correct data division; after generating the training and test sets, they can be saved as new data packets.

[0025] In the embodiments of the present application, the unified format and cleaning ensure the consistency and reliability of the data, allowing it to flow smoothly in subsequent processes, and the data cleaning and standardization effectively remove incorrect and inconsistent data, improving data quality and providing accurate information for further analysis and prediction. Through feature selection and dimensionality reduction, the complexity of model training is reduced, the risk of overfitting is reduced, and the efficiency and prediction ability of the model are improved. By generating data packets that meet the cross-network transmission standards, the data can be quickly and securely transmitted in different network environments, supporting data interaction between systems, removing redundant features and reducing the dimensionality of the operation, reducing the storage and computing burden, and speeding up data processing and model training.

[0026] In a preferred embodiment of the present application, the data packet is transmitted from the first network environment to the second network environment through a preset cross-network communication method, including: The data packet is prepared and packaged to obtain a packaged data packet, specifically including: preparing the data, the preparation process including processing missing data using a fill or delete strategy, scaling the data as needed, ensuring that the feature values are within the same range, converting the classification features to numerical types, selecting features meaningful for model training, and removing redundant and irrelevant features; the processed data is converted into a format that facilitates model training, storage, transmission and sharing, and the converted data is packaged to obtain a data packet; The packaged data packet is checked to ensure the accuracy of the packaged data transmission process to obtain accurate data, specifically including: when the data is packaged, the hash value of the data is calculated and recorded, and the receiver recalculates the hash value of the received data during the data transmission process and compares it with the hash value of the sender; before the data is packaged and saved, the hash value of the data packet is calculated for subsequent verification; after receiving the data, the hash value of the received data is recalculated and compared with the transmitted hash value to ensure that the data has not been tampered with during transmission; if the hash values of the data are found to be inconsistent during the verification process, the data has been damaged or tampered with during transmission, and the data can be requested to be resent; consistency check is performed on the data to ensure consistency during transmission; The accurate data is transmitted through a cross-network communication method, specifically including: selecting a suitable network protocol for data transmission according to different needs, to ensure the accuracy of the transmitted data, serializing the data for verification, serializing the data into JSON format for cross-network transmission, calculating the hash value of the data to ensure that the data has not been tampered with during transmission; the data is packaged and encrypted to improve the security of data transmission; The cross-network method from the first network environment to the second network environment is evaluated, specifically including: during the cross-network transmission process, the target needs to be clear, from the first network environment to the second network environment; evaluate different cross-network transmission methods to determine which one is suitable for the two network environments; the transmission speed, bandwidth utilization, security, reliability, stability and cost can be evaluated; The reliability and integrity of the cross-network communication method are evaluated to obtain an accurate communication method, specifically including: ensuring that network communication can be transmitted stably and efficiently, preventing data loss, delay and terminal, ensuring that data in cross-network communication is not tampered with, lost or damaged during transmission, and can receive complete data packets, selecting several common communication methods, evaluating the reliability and integrity of the cross-network communication method, which can be evaluated by the following evaluation standards: Whether the network link is stable, the probability of connection interruption and disconnection; the amount of data packets lost during data transmission; the time delay from data sending to receiving; the number of data packets that need to be resent due to data loss or error; Whether the data transmission can ensure that the data is not tampered with and lost during transmission; using data verification technology to check data integrity; ensuring that the data of the sending end and the receiving end is consistent, avoiding data damage; According to the characteristics and needs of the network environment, select the appropriate cross-network communication mode, and configure the performance indicators of the communication mode, configure the communication mode, and ensure that the cross-network data can be transmitted correctly; Ensure that the network device configuration of the communication mode is correct, including: according to the evaluated cross-network communication mode, determine the device, common devices include: router, firewall, switch, VPN gateway and SD-WAN device; Ensure that the router supports the protocol function of cross-network communication; Configure the VLAN and link aggregation of the switch to ensure smooth communication traffic; Configure firewall rules to allow or block specific traffic according to the needs of cross-network communication; Ensure that only necessary ports and protocols are allowed to pass through the firewall to prevent external attacks or data leakage; If the communication mode is based on VPN, configure the VPN gateway to ensure that data can be transmitted securely through an encrypted channel; Check network connectivity and verify network device configuration; Transmit the data packet from the first network environment to the second network environment through the cross-network communication mode, including: through the selected cross-network communication mode, the data packet is circulated a number of times to ensure that the data packet can be correctly transmitted between the first network and the second network; Configure static routes on the router of the first network pointing to the second network to ensure the security of the data packet, and configure firewall rules on both ends to allow relevant traffic to enter and exit; Cross-network communication needs to be encrypted, configure VPN tunnel to ensure the safe transmission of data packets between networks; After completing all configurations, ensure that the data packet can be correctly transmitted from the first network to the second network without data loss or excessive delay and other problems.

[0027] In the embodiment of the application, the cross-network transmission of the data packet from the first network environment to the second network environment is stable, accurate and secure. The beneficial effects of each step improve the efficiency of data transmission, reduce errors, ensure that the network environment is correctly configured, and select the most suitable communication mode to improve the reliability, integrity and security of the overall system. These measures effectively ensure the seamless, efficient and secure transmission of data.

[0028] In a preferred embodiment of the application, an optimization algorithm is used to update the data, and the optimization algorithm uses an incremental update algorithm, including: Pull transmission data supporting incremental collection from the communication mode, specifically including: by recording the last collection timestamp, only pulling newly added or modified data after the timestamp, and by unique identifier to pull newly added or modified data; according to the communication architecture and data transmission mode of the system, select the appropriate data transmission protocol; the specific steps of incremental pulling are: after each successful data collection, update the records in the system for next time; according to the query mode supported by the system, use the timestamp as the filtering condition to obtain the newly added or modified data; according to the protocol specification, parse and process the returned data; if data is missed due to reasons, the missing data needs to be pulled again to ensure that the network is interrupted or the communication is abnormal, and to avoid missing data; According to the collected transmission data, select an update model for calculation, specifically including: determine which models need to be updated, such as: linear regression model, statistical model and business rule model; select the appropriate model for update strategy, common selection methods: incremental learning-based model, periodic full update model and hybrid method; to ensure that the collected transmission data can effectively update the model, the data needs to be preprocessed; select the incremental update model, which will be trained according to the new transmission data, gradually adjusting the weight or parameter of the model; Adjust the model parameters by calculating the increment, specifically including: determine how to adjust the model parameters according to the incremental data, the incremental data refers to the collected data added on the basis of the original training data; incremental update refers to the model parameters or weights will be adjusted based on incremental data every time new data comes; select the appropriate model to support incremental update, incremental update needs to preprocess the data, standardize the data and select the features; Calculate the gradient required for data increase and update by incremental learning method, specifically including: incremental learning method allows the model to be updated gradually when new data comes, without the need to retrain the entire model, in linear regression, the gradient is the partial derivative of the loss function with respect to the model parameters. In the incremental learning process, we need to calculate the gradient of the loss function with respect to the model parameters every time new data comes, and then update the model parameters using the gradient; by receiving incremental data, the model is trained gradually, for each new sample or data batch, calculate the gradient of the loss function, use gradient descent or other optimization methods to update the model parameters based on the calculated gradient; select the stochastic gradient descent method to support incremental update algorithm; According to the calculated incremental update model parameters, specifically including: in incremental learning, the target is to calculate the corresponding gradient and update the parameters of the model every time new data is received; preprocessing the received data to extract features; through a linear regression model, the model updates its parameters using an incremental learning method to process new communication record data; by using a step-by-step updating method, each time a new communication record is input, the gradient is calculated and the model parameters are updated; According to the calculated incremental update model parameters, the new data is integrated, specifically including: preprocessing the original text data, including removing stop words, punctuation, etc., and then converting it into numerical features for the model to process; each time new data is transmitted, the model can calculate and adjust the current model parameters, using online learning to integrate new data into the existing model, avoiding retraining the entire model; first, initialize the linear regression model and set the initial parameters; using the current model and the new data, calculate the predicted value and error of the model, and then calculate the gradient based on the error; combine the gradient of the new data with the current model parameters to obtain the updated model parameters; according to the calculated gradient and learning rate, update the model parameters so that the model better fits the new data; after the new data is added, use the updated model for inference and prediction to process new communication records in the future; In the incremental update process, the newly added data is optimized and adjusted, specifically including: converting text data for model training features, which requires noise removal, word segmentation, and word vectorization processing on text data; initialize the model parameters and learning rate, calculate the gradient based on the current model and new data, which represents the influence of the model error on the model parameters, adjust the model parameters according to the calculated gradient, usually using gradient descent method to update; according to the current state of the model and the new data, calculate the loss and gradient brought by the new data; update the model parameters according to the calculated gradient.

[0029] In the embodiments of the present application, through incremental updating, the model can quickly adapt to changes in new data, avoiding long retraining, saving computing resources, improving the real-time performance of data updating, and making the data updating process more efficient through incremental collection and incremental learning, reducing unnecessary data transmission and computing burden, reducing system burden, and through dynamic adjustment of model parameters, incremental updating enables the model to respond to data changes at any time, maintaining its prediction accuracy and business value. Compared with traditional full update, incremental update avoids repeated calculation and excessive training, thereby improving the running efficiency of the system, optimizing the process of new data to ensure the effectiveness of the data, improving the usability of the data, and thus ensuring that the training results of the model are not disturbed by low-quality data.

[0030] In a preferred embodiment of the present application, the pre-processed data is processed and updated in the second network environment to obtain updated data, comprising: The data in the second network environment is pre-processed to obtain pre-processed data, specifically including: pre-processing the data obtained from the second network environment to ensure that the data is free of noise and consistent in format; converting the data, the converted data being suitable for a linear regression model, the conversion method including standardization and normalization, time data processing and category data coding; predicting the linear regression model, selecting features of the data, removing redundant features, and creating new features from existing features; when performing data analysis, the data needs to be divided into a training set and a test set, and the processed data can be saved; The pre-processed data is format-verified to verify whether the data type conforms to the data model to obtain complete data, specifically including: ensuring that the data type of each field conforms to the requirements of the predetermined data model, and for columns whose data type does not conform to the requirements, conversion can be performed; range checking is performed on numerical data, and if it exceeds a reasonable range, it can be corrected; if the data does not exist within a valid range, it can be processed; in addition to checking the data type and range, the data format also needs to be verified; verify whether the character string column conforms to the predetermined format, and for columns containing date and time, ensure that the date format conforms to the requirements; check whether there is control and process according to the rules; perform consistency checking on the data to ensure that the logical relationship and rules of the data are followed; the verified data is saved; The complete data is compared with target data in the second network environment to determine the consistency of the data, specifically including: obtaining target data from the second network environment, ensuring that the target data obtained from the second network environment is pre-processed consistently with the local data before data comparison; in order to ensure data consistency, comparison rules need to be set, including: selecting comparison fields, determining comparison formats and data comparison; completely comparing the corresponding columns in the two sets of data, if the two sets of data are not completely identical, the difference can be output for further processing; ensure that there is no data missing that causes the comparison to fail; check whether the data meets the consistency requirements, if differences are detected in the data, processing can be performed according to business logic; according to business rules, the local data can be synchronized with the target data to ensure that the target data is consistent with the complete data; The complete data is updated to obtain updated metadata, specifically including: updating the obtained data, the update content including: modifying existing records, adding new records and deleting unnecessary records; if modifying existing records, the fields that need to be changed can be updated by comparing data; if there are new data to be added to the existing data, the new data and the old data can be merged; if some records need to be deleted, the deletion operation can be performed; metadata is data about data, including data structure, field description and update time; updating the data dictionary includes the name, type and description information of each field; the record update time is very important to ensure the accuracy of data and metadata; after updating the data and metadata, the updated data is selected to be saved to the database; Performing sentiment analysis on the updated metadata to generate sentiment labels, specifically including: obtaining the updated metadata, analyzing the communication records through sentiment analysis tools, classifying the communication records into different sentiment labels, including positive, negative and neutral; according to the sentiment score range, different labels can be set; visualization can be used to help understand the distribution of emotions in the data; each communication record is assigned a sentiment label, which can be combined into metadata and processed; Extracting keywords in sentiment labels and storing them by using NLP technology to obtain metadata fields, specifically including: extracting keywords related to sentiment labels from communication records through NLP technology, and the technology for extracting keywords uses TF-IDF; add the extracted sentiment labels and keywords to the metadata; save the final metadata to the database; Updating the metadata fields to obtain updated data, specifically including: according to the updated sentiment analysis algorithm, the sentiment label of each record needs to be re-evaluated; new keywords need to be extracted based on new analysis methods or changes in sentiment labels; in addition to sentiment labels and keywords, other metadata fields may also need to be added; add a new field to each comment to represent the character length of the comment as additional metadata; after all metadata fields are updated, the updated data can be saved to the database.

[0031] In the embodiment of the present application, data preprocessing and format verification improve the accuracy and consistency of data, reduce data errors, improve data quality, ensure that subsequent analysis can be based on high-quality input, ensure the correctness and integrity of data in the target environment through comparison and consistency check, thereby increasing the credibility of the data, and the sentiment analysis adds emotional insights to the data, so that the data is not limited to quantitative analysis, but can also be analyzed in depth from the perspective of emotional tendency and emotional fluctuation, by extracting and storing emotional labels and keywords, the update of metadata makes data management more orderly, and provides convenience for quick retrieval and analysis, and the updated metadata and emotional labels help to more accurately understand the emotional feedback of customers or users, and can provide effective support for market analysis, customer support and decision-making.

[0032] In a preferred embodiment of the present application, the updated data is returned to the first network environment through a cross-network communication method, and the updated data is applied in the first network environment, including: The cross-network communication method is safe and reliable, and a firewall is configured for the cross-network communication method, specifically including: cross-network communication generally refers to data exchange between different networks, which may include Internet, intranet and external service connection, first, we need to ensure that the selected communication method can effectively prevent man-in-the-middle attacks, data leakage and other network attacks; when cross-network communication, it is necessary to design a secure network topology first, to ensure the security of data transmission, and to encrypt the data using appropriate encryption technology; the firewall can be used to manage and control cross-network communication traffic; A secret key is configured for the cross-network communication method, and information is desensitized to obtain secure data, specifically including: using a pair of public and private keys for encryption and decryption operations, exchanging secret keys in a secure manner, common methods include encrypting through public keys or using TLS protocol for key exchange; ensure that the AES secret key is securely stored in the devices of both parties; desensitization ensures that sensitive data will not leak the actual content during cross-network transmission, protecting user privacy and business secrets; common desensitization techniques include replacement, encryption and data deletion; hide some parts of the data and only display the necessary information; replace sensitive information with non-real, meaningless data; encrypt sensitive data to ensure that even if the data is leaked, secure data is obtained; The security data is encapsulated to obtain encapsulated data, specifically including: ensuring the confidentiality of the data in the transmission process by using an encryption algorithm; using digital signature to ensure the integrity and identity verification of the data; packaging the encrypted data with signature information, timestamp and other additional information; selecting a suitable encapsulation format for data encapsulation; before encapsulating the data, the data needs to be encrypted, and encryption ensures the confidentiality of the data in the transmission process; using AES algorithm to encrypt the data; in order to ensure that the data is not tampered with in the transmission process, the encrypted data is signed using digital signature technology; the encrypted data, signature information and other additional information are encapsulated into a complete data packet together; The encapsulated data is securely transmitted from the second network environment to the first network environment to obtain transmission data, specifically including: ensuring that the data content will not be stolen or leaked in the network transmission process, ensuring the identity verification of the sender and the receiver, preventing man-in-the-middle attack, ensuring that the data is not tampered with in the transmission process, and preventing attackers from replaying old transmission data; during the transmission process, the goal is to ensure the integrity of the data in the transmission process; the receiver in the first network environment needs to verify the signature of the data to ensure that the data has not been tampered with, and the receiver will use the public key to verify the signature of the received data; once the signature verification is successful, the receiver will continue to decrypt the data to recover the original sensitive information; in order to prevent attackers from replaying old data packets, a timestamp and a random value are usually used in the transmission, and the receiver will verify the timestamp to ensure the timeliness of the data packet; once the data has been verified through the above steps and no tampering or replay attack has been found, the receiver can safely process the data; The transmitted data is received in the first network environment to obtain received data, specifically including: in the first network environment, a server end needs to be prepared to receive the transmitted data; the server end needs to start the HTTPS service and listen to the specified port to receive the data; the server end will send the data to the server in the first network environment through the HTTPS protocol; the data containing encrypted data, signature and timestamp is received through the POST request; the signature is verified using the public key to ensure that the data has not been tampered with; the data is decrypted using the symmetric key to recover the original content; ensure that the data has not been replayed, and verify the validity of the timestamp; once the data is verified, it can be stored or further processed; The received data is verified to ensure the integrity of the data, including: the integrity of the data is the key to ensure that the data is not tampered with or lost during transmission; the sender calculates and attaches the hash value of the data, and the receiver recalculates the hash value by the same algorithm and compares it; the sender signs the hash value of the data, and the receiver verifies the signature with the public key to ensure that the data has not been tampered with; the sender calculates the hash value of the data and attaches it to the data, and the receiver recalculates the hash value after receiving the data and verifies whether they are consistent; after receiving the data, the receiver first extracts the hash value from the data, then recalculates the hash value of the data, and compares it with the attached hash value to confirm that the data has not been tampered with during transmission; if a digital signature is used for verification, the sender will generate a signature for the data, and the receiver will use the sender's public key to verify the signature to ensure that the data has not been tampered with during transmission; to prevent replay attacks, the receiver can also use a timestamp or a unique identifier to ensure the timeliness of the data; the receiver can check the timestamp after receiving the data to ensure that the data is within the valid period and prevent replay attacks; if all verifications are passed, the receiver can confirm the integrity of the data; The complete data is format checked to verify the correctness of the format, including: whether the data is of the expected type; whether the data conforms to the predetermined format; whether the data contains all necessary fields; whether the data conforms to the expected range or length; before format checking, we need to define the standard of data format, such as: field name, field type, required field, specific format and field length range; check if the data contains all required fields; verify the data type of each field; if some fields have length restrictions, the length of the field can be verified; if the date field has specific range requirements, the date can be verified to be within the valid range; the above verification steps are integrated into a complete data format checking process; The complete data is merged with the update data into the first network environment and stored, including: data merging operation to ensure that new updates do not overwrite existing data, and the merged data is complete and consistent; data storage in appropriate databases or data storage systems to ensure that the data can be accessed and used by the first network environment; if the update data contains corresponding fields, the corresponding fields in the complete data are overwritten by the update data; if the update data contains new fields, these new fields should be merged into the complete data; the part of the complete data that does not exist in the update data field should be retained; when merging the complete data and the update data, the following rules can be followed: if the update data contains existing fields, overwrite the corresponding fields in the complete data; if the update data contains new fields, add these new fields to the complete data; in this way, the update data is successfully merged into the complete data; after merging the data, the result needs to be stored in the designated storage system in the first network environment.

[0033] In the embodiment of the present application, the firewall measure key is added to ensure the security of the communication process in the cross-network transmission, prevent data privacy leakage and security tampering, ensure the integrity of the transmitted data, and correctly receive the data without error, avoid incorrect parsing, and correctly parse the complete data based on the latest complete data and the complete updated data to ensure the support of the data operation.

[0034] As shown in Figure 2 The embodiment of the present application also provides a cross-network program data updating system, which comprises: An acquisition module, which acquires the to-be-updated data from a data source in at least one first network environment; A processing module, which pre-processes the to-be-updated data to generate a data packet conforming to a cross-network transmission standard; A calculation module, which transmits the data packet from the first network environment to a second network environment through a preset cross-network communication mode, and adopts an optimization algorithm to update the data, wherein the optimization algorithm adopts an incremental updating algorithm, and the specific steps are as follows: Acquire the to-be-updated data from the first network environment; Convert the to-be-updated data into a format supported by the second network environment; Perform integrity check and encryption processing on the converted data to obtain encrypted data; Decrypt the encrypted data, and calculate the difference between the to-be-updated data and the last transmission data by using a difference algorithm to obtain incremental data; Add a mark to the incremental data, and transmit the incremental data with the mark from the first network environment to the second network environment through the cross-network communication mode; An updating module, which processes and updates the pre-processed data in the second network environment to obtain updated data; A return transmission module, which returns the updated data to the first network environment through the cross-network communication mode; An application module, which applies the updated data in the first network environment.

[0035] It is explained that the system corresponds to the above method, and all implementation manners in the above method embodiment are applicable to this embodiment, and the same technical effects can be achieved.

[0036] The above is the preferred embodiment of the present application. It should be noted that, for ordinary skilled persons in the technical field, without departing from the principles of the present application, a number of improvements and refinements can be made, which should also be considered as the protection scope of the present application.

Claims

1. A cross-network program data updating method, characterized by, The application comprises the following steps: Obtaining the data to be updated from the data source in at least one first network environment; Preprocessing the data to be updated to generate data packets conforming to cross-network transmission standards; Obtaining the data to be updated from the first network environment; Converting the data to be updated into a format supported by the second network environment; Performing integrity check and encryption processing on the converted data to obtain encrypted data; Decrypting the encrypted data and calculating the difference between the data to be updated and the last transmission data by using a difference algorithm to obtain incremental data; Adding a mark to the incremental data and transmitting the marked incremental data from the first network environment to the second network environment through cross-network communication; Processing and updating the preprocessed data in the second network environment to obtain updated data; Returning the updated data to the first network environment through cross-network communication; Applying the updated data in the first network environment.

2. The cross-network program data update method of claim 1, wherein, Obtaining the data to be updated from the data source in at least one first network environment comprises the following steps: Determining the range of data to be updated from the database; Narrowing down the data source of the data to be updated; Determining that the data source is suitable for the first network environment; Establishing a network connection between the data source and the first network environment; Transmitting the data source to the first network environment through the network connection to obtain the data to be updated.

3. The cross-network program data update method of claim 2, wherein, The preprocessing of the data to be updated to generate data packets conforming to cross-network transmission standards comprises the following steps: Collecting the data to be updated, and determining that the collected data has a unified format; Cleaning the collected data to obtain accurate data; Standardizing and converting the accurate data to obtain converted data; Fusing the converted data to obtain a unified data structure; Predicting the unified data to generate new data features; Statistically analyzing the new data features, selecting relevant features of the data, and removing redundant features and reducing the dimension of the data; Dividing the data into a training set and a test set to obtain new data packets.

4. The cross-network program data update method of claim 3, wherein, Transmitting the data packets from the first network environment to the second network environment through a preset cross-network communication method comprises the following steps: Preparing and packaging the data packets to obtain packaged data packets; Verifying the packaged data packets to ensure the accuracy of the data during transmission to obtain accurate data; Transmitting the accurate data through cross-network communication; Evaluating the cross-network method of transmitting from the first network environment to the second network environment; Evaluating the reliability and integrity of the cross-network communication method to obtain an accurate communication method; Ensuring that the network device configuration of the communication method is correct; Transmitting the data packets from the first network environment to the second network environment through the cross-network communication method.

5. The cross-network program data update method of claim 4, wherein, An optimization algorithm is used to update the data, and the optimization algorithm uses an incremental update algorithm, comprising the following steps: Pulling transmission data supporting incremental collection from the communication method; Selecting an update model for calculation according to the collected transmission data; Adjusting the model parameters by calculating the increment; Calculating the gradient required for data update by incremental learning; Updating the model parameters according to the calculated increment; Integrating the calculated incremental model parameters into newly added data; In the incremental update process, the newly added data is optimized and adjusted.

6. The cross-network program data update method of claim 5, wherein, The preprocessed data in the second network environment is processed and updated to obtain updated data, including: The data in the second network environment is preprocessed to obtain preprocessed data; The preprocessed data is format-verified to verify whether the data type conforms to the data model to obtain complete data; The complete data is compared with the target data in the second network environment to determine the consistency of the data; The complete data is updated to obtain updated metadata; The updated metadata is subjected to sentiment analysis to generate sentiment labels; Key words in the sentiment labels are extracted and stored by using NLP technology to obtain metadata fields; The metadata fields are updated to obtain updated data.

7. The cross-network program data update method of claim 6, wherein, The updated data is returned to the first network environment through cross-network communication, and the updated data is applied in the first network environment, including: Determine that the cross-network communication is safe and reliable, and configure a firewall for the cross-network communication; A secret key is configured for the cross-network communication, and the information is desensitized to obtain secure data; The secure data is encapsulated to obtain encapsulated data; The encapsulated data is securely transmitted from the second network environment to the first network environment to obtain transmission data; The transmitted data is received in the first network environment to obtain received data; The received data is verified to verify the integrity of the data to obtain complete data; The complete data is format-verified to verify the correctness of the format; The complete data is merged with the updated data in the first network environment and stored.

8. A cross-network program data updating system characterized by comprising: Applied in the method of any one of claims 1-7, including: An acquisition module acquires the data to be updated from a data source in at least one first network environment; A processing module pre-processes the data to be updated to generate a data packet conforming to a cross-network transmission standard; A calculation module transmits the data packet from the first network environment to the second network environment through a preset cross-network communication, and uses an optimization algorithm to update the data, wherein the optimization algorithm uses an incremental update algorithm, and the specific steps are as follows: Acquire the data to be updated from the first network environment; Convert the data to be updated into a format supported by the second network environment; Perform integrity verification and encryption processing on the converted data to obtain encrypted data; Decrypt the encrypted data and calculate the difference between the data to be updated and the last transmission data by using a difference algorithm to obtain incremental data; Add a mark to the incremental data, and transmit the incremental data with the mark from the first network environment to the second network environment through cross-network communication; An update module processes and updates the preprocessed data in the second network environment to obtain updated data; A return transmission module returns the updated data to the first network environment through cross-network communication; An application module applies the updated data in the first network environment.

9. A computing device, comprising: Including: One or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of claim 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a program, and the program is executed by the processor to implement the method in claim 7.

Images