Data pushing method, system and equipment for non-confidential data and medium
Through the data push method for non-confidential data, the problem of low data exchange efficiency and difficult security in cross-company product and service docking is solved, and efficient and secure data push and synchronization is achieved.
Patent Information
- Application Number
- CN202411782483.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-05
- Publication Date
- 2025-05-06
AI Technical Summary
In the process of cross-company product and service docking and system integration, it is difficult for the existing technology to efficiently realize data exchange of non-confidential data, resulting in difficulty in repeated development, data synchronization and data security.
It provides a data push method for non-confidential data. By carefully configuring data sources, generating JSON strings, introducing data signature and verification mechanisms, and supporting various push processes such as timed push and incremental push, ensuring the integrity and authenticity of data during transmission.
It realizes end-to-end timed push and incremental push of non-confidential data, reduces development workload, improves the efficiency and security of data synchronization, and is suitable for the data receiving side in different network environments.
Smart Images
Figure CN119939013A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of data processing, and more specifically relates to a data push method, system, device and medium for non-confidential data. Background Art
[0002] In the process of traditional cross-company product service docking and system integration, there are scenarios for basic business data information docking. Confidential data is usually processed by corresponding specifications and professional software and hardware. For non-confidential data, if the docking method of confidential data is also used, it will reduce the efficiency of data exchange and cause a waste of encryption resources. In order to achieve the docking requirements of such data, it is usually necessary to carry out secondary development for the docking products to realize data exchange.
[0003] Take single sign-on docking as an example: during the docking process, usually only the main system user center is retained as the functional center for actual user login control. The user center of other systems only performs login verification for their own systems. Therefore, during the synchronization process, such data usually adopts the principle of minimizing demand synchronization, that is, only limited user information is transmitted, and confidential data (password-related data, user personal information) is usually not within the transmission scope. At the same time, because it involves login verification-related data, the data must also be tamper-proof and anti-forgery when it is transmitted.
[0004] Currently, there are several problems in the data synchronization process that need to be solved: 1. In the docking process, most of the docking logic is roughly the same, but because the target service of each docking is different, each development can rarely be reused. Each docking will involve a lot of repetitive work.
[0005] Second, for security reasons, the databases of most services are prohibited from being exposed outside the internal network of the service group. In large-scale software integration system projects, multiple services may be deployed in different network segments or even in different network environments. The database-to-database docking mode of both parties is difficult to achieve in this case. The basic functions of the currently commonly used general data synchronization tools are mostly to synchronize data in a database-to-database mode. The API docking mode requires the receiving end to provide relevant API interfaces. It also brings additional development workload.
[0006] 3. Since the two parties are not in the same system, many data synchronization processes are difficult to synchronize, which can easily lead to problems such as duplicate data distribution and data omission.
[0007] Therefore, there is a need for a data exchange tool that can selectively push data and push incrementally, is plug-and-play and does not require additional development and adaptation, supports scheduled push, can be connected through interfaces, and supports data signatures. Summary of the invention
[0008] In response to the above problems, the purpose of the present invention is to provide a data push method, system, device and medium for non-confidential data, which realizes end-to-end scheduled push and incremental push of non-confidential data, while ensuring the data integrity and authenticity during the data push process.
[0009] In order to achieve the above object, the present invention is implemented through the following technical solutions: In a first aspect, an embodiment of the present application provides a data push method for non-confidential data, including: Configure the data source for the data push end; Configure the data source for the data receiving end and lock the configuration information; The data push terminal collects data from the data source, processes the data, and generates a json string as the push data; Sign the pushed data through the data push end and configure the data receiving end; Execute the corresponding push process according to the push demand to send the push data to the data receiving end; the push process includes: one-time push process, scheduled push process, full push process and incremental push process; Obtain the pushed data through the data receiving end and perform data signature verification; After the data is verified, the pushed data is stored in the data receiving end.
[0010] In an optional implementation, configuring a data source for the data push end includes: Preload the database JDBC package on the data push end and introduce it as an expansion of the data package when the program starts; Configure the database type, URL address, port, user name, password, database name, and mode name of the data source for the data push end; Store passwords in reversibly encrypted form.
[0011] In an optional implementation, configuring a data source for the data receiving end and locking the configuration information includes: Configure a unique data push terminal for the data receiving terminal as the data source of the data receiving terminal; Configure related configuration items in the visual configuration interface of the data receiving end and lock the configuration information.
[0012] In an optional implementation, the data push end collects data from the data source, processes the data, and generates a json string as the push data, including: Connect to the database of the data source through the data push terminal, and collect field information in the specified table according to the pre-configured collection fields and field aliases; Perform value replacement and constant insertion on field information; Convert the processed field information into a JSON string as the push data.
[0013] In an optional implementation, the data push end performs data signing on the pushed data and configures the data receiving end, including: Randomly generate an RSA key pair, generate a hash summary based on the pushed data, sign the hash summary with the private key, and generate a signature data string; Configure the address and port of the data receiving end for the data push end, and store the configuration information in the database.
[0014] In an optional implementation, the push process includes: directly calling a push function to complete a data push for the push data; The timed push process includes: using multi-threading and TimerTask to generate a timed push task, configuring the timed push information and recording the configuration information; when the timed push task is started, generating a corresponding working thread, and configuring the execution frequency through TimerTask.
[0015] In an optional implementation, the full push process includes: checking the data volume of the push data, reading the push data in groups according to the data volume and the preset system configuration, and pushing the data to the data receiving end; The incremental push process adopts the auto-increment field check incremental push mode or the data incremental push mode with auxiliary tables and triggers; The auto-increment field check incremental push mode includes: specifying the auto-increment field in the push data; the auto-increment field includes the auto-increment primary key, update time or creation time; after completing the first push of the push data, recording the largest auto-increment field in the data pushed this time as a reference value, and checking the push based on the reference value when the next push of data starts; The data incremental push mode with auxiliary tables and triggers includes: Collect the primary key of the pushed data and allow the preset push program to create new tables and triggers in the source library; Check whether the incremental push task of the preset push program's own service library has been executed; If not executed, perform a full push first, create a source table in the source database, add a trigger to the source table, and configure the change type for incremental push; If full push has been executed, perform incremental push, obtain the primary key of the unsynchronized push data according to the source table, search for data in the source table and synchronize it.
[0016] In an optional implementation, the acquiring of the pushed data through the data receiving end and performing data signature verification includes: Obtain push data and key pair through the data receiving end; Based on the key pair, import the signature public key and verify the signature data string based on the RSA or SM2 algorithm.
[0017] In an optional embodiment, the method further comprises: After the data push end completes the data signature of the pushed data, a first test sample is generated according to the current configuration information to simulate the data style of the pushed data; When the data receiving end completes the storage operation of pushing data, the second test sample is saved according to the current configuration information; The first test sample is compared with the second test sample, and configuration information of the data push end and the data receiving end is determined according to the comparison result.
[0018] In a second aspect, the embodiment of the present application further provides a data push system for non-confidential data, including: The system comprises: a data push end and a data receiving end, wherein the data push end collects and pushes data and sends the data to the data receiving end through a network; The data push end includes: Push configuration module, used to configure the data source for the data push end; The data collection and processing module is used to collect data from the data source, process the data and generate a json string as the push data; Data signature module, used to sign the pushed data and configure the data receiving end; The data push module is used to execute the corresponding push process according to the push demand to send the push data to the data receiving end; the push process includes: one-time push process, scheduled push process, full push process and incremental push process; The data receiving end comprises: A receiving configuration module is used to configure the data source for the data receiving end and lock the configuration information; Data acquisition and signature verification module, used to obtain pushed data and perform data signature verification; The data storage module is used to store the pushed data after the data verification is passed.
[0019] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, the steps of the data push method for non-confidential data as described in any one of the above items are implemented.
[0020] In a fourth aspect, an embodiment of the present application further provides a storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of data push for non-confidential data as described in any of the above items are implemented.
[0021] It can be seen from the above technical solutions that the present invention has the following advantages: In the data push method for non-confidential data provided by the present application, firstly, by finely configuring the data source, ensure that the connection between the data push end and the receiving end is stable and reliable. In the data processing stage, the method can flexibly convert the information in the data source into a JSON string, which provides great convenience for the subsequent processing of the data. At the same time, by introducing the data signature and signature verification mechanism, the integrity and authenticity of the data during the transmission process are effectively guaranteed, and the risk of data tampering is prevented. In addition, the method also provides a variety of push processes to meet the data push requirements in different scenarios. In terms of configuration and testing, the method also performs well, ensuring the reliability and ease of use of the system. In summary, the method is an efficient, safe and reliable data push solution with broad application prospects.
[0022] This application provides a relatively general data synchronization solution for non-confidential, small and medium-sized batch data, which can effectively reduce the workload when connecting cross-product business.
[0023] Compared with general ETL tools, this application is lighter and easier to use in related synchronization work. At the same time, it comes with a built-in data signature function to ensure data reliability without secondary development.
[0024] This application provides an API access mode for the data receiving end, which is suitable for use in scenarios where the database is prohibited from being directly exposed outside the business network.
[0025] This application provides an incremental push solution that can reduce repeated data push and avoid data omission as much as possible. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0027] Figure 1 A flowchart of a data push method for non-confidential data provided in this application.
[0028] Figure 2 A flowchart of the auto-increment field check incremental push mode provided for this application.
[0029] Figure 3 A flowchart of the data incremental push mode with auxiliary tables and triggers provided for this application.
[0030] Figure 4 A schematic diagram of the structure of the data push system for non-confidential data provided in this application.
[0031] Figure 5 A schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION
[0032] In the specific steps of the data push method for non-confidential data described in detail below, various embodiments of the present disclosure will be described more comprehensively. The present disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of the present disclosure to the specific embodiments disclosed herein, but the present disclosure should be understood to cover all adjustments, equivalents and / or alternatives that fall within the spirit and scope of the various embodiments of the present disclosure.
[0033] Hereinafter, the terms "include" or "may include" used in various embodiments of the present disclosure indicate the presence of disclosed functions, operations, or elements, and do not limit the addition of one or more functions, operations, or elements. In addition, as used in various embodiments of the present disclosure, the terms "include", "have", and their cognates are intended only to indicate specific features, numbers, steps, operations, elements, components, or a combination of the foregoing, and should not be understood as first excluding the presence of one or more other features, numbers, steps, operations, elements, components, or a combination of the foregoing or the possibility of adding one or more features, numbers, steps, operations, elements, components, or a combination of the foregoing.
[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0035] See also Figure 1 The figure is a method flow chart of a data push method for non-confidential data in a specific embodiment, the method comprising: S1: Configure the data source for the data push end.
[0036] In a specific implementation, the JDBC package of the database is first preloaded on the data push end and introduced as an extension of the data package when the program starts; then the database type, URL address, port, user name, password, database name, and mode name of the data source are configured for the data push end; finally, the password is reversibly encrypted and stored.
[0037] For example, configuring data sources involves adding, deleting, modifying, and listing data sources. In addition, the JDBC packages of common databases on the market (mysql, pg, oracle, including some domestic databases) are preloaded in the program package, and the expansion and introduction of data packages are supported when the program is started.
[0038] The specific configuration data source library includes the database type, URL address, port, user name, password, database name, mode name (optional) and other basic database connection information. The password should be stored in encrypted form, and since the password will be used for subsequent database connections, the password is reversibly encrypted. Passwords cannot be viewed when editing a data source.
[0039] S2: Configure the data source for the data receiving end and lock the configuration information.
[0040] In a specific implementation, a unique data push terminal is configured for the data receiving terminal as the data source of the data receiving terminal; relevant configuration items are configured in the visual configuration interface of the data receiving terminal, and the configuration information is locked.
[0041] For example, a receiving end only supports configuring one data source responsible for receiving data. The data receiving end provides a simple visual configuration interface (BS architecture), which can be controlled and closed in the visual interface or the jar package startup configuration file. At the same time, for the configuration items configured through the front-end interface, you can choose to lock the configuration after completing the configuration. After locking the configuration, the front-end configuration interface will also be closed. Once the configuration is locked, it cannot be accessed through the front-end configuration or other methods. It can only be unlocked by deleting the lock file in the server where the jar package is located.
[0042] S3: Collect data from the data source through the data push end, process the data and generate a json string as the push data.
[0043] In a specific implementation, firstly, the database of the data source is connected through the data push end, and the field information in the specified table is collected according to the pre-configured collection fields and field aliases. Then, the field information is value replaced and constants are inserted. Finally, the processed field information is converted into a json string as the push data.
[0044] For example, after the data push end is connected to the database, it can collect field information in the specified table. The collected fields can be configured as needed, and field aliases can be specified. It supports simple data processing functions such as value replacement and constant insertion. Finally, after the data processing is completed, a json string that can be submitted to the data receiving end is generated.
[0045] S4: The data push end signs the pushed data and configures the data receiving end.
[0046] In a specific implementation, firstly, an RSA key pair is randomly generated, a hash summary is generated based on the pushed data, the hash summary is signed using a private key, and a signature data string is generated; then, the address and port of the data receiving end are configured for the data push end, and the configuration information is stored in a database.
[0047] For example, the data push terminal provides a data signature function that can randomly generate an RSA key pair. After the key pair is generated, the private key is not directly displayed, but the public key can be viewed at any time. The data signature process is as follows: hash the sent data, and use the private key to sign the summary content to generate a signature data string.
[0048] After the data signature is completed, configure the data receiving end address, port and other information. The configured information is stored in the database.
[0049] S5: Execute the corresponding push process according to the push demand to send the push data to the data receiving end.
[0050] The push process includes: a one-time push process, a scheduled push process, a full push process and an incremental push process.
[0051] Specifically, a one-time push directly calls the push function to complete a data push.
[0052] Scheduled push uses multithreading + TimerTask to implement scheduled tasks. After configuration is completed, only the configuration is recorded, and the scheduled task thread is not actually generated. After clicking Start, a new working thread is generated, and the execution frequency is configured through the relevant configuration of TimerTask.
[0053] When pushing the full data, first check the data volume, then read and push the data in groups based on the data volume and system configuration.
[0054] In addition, the data push end supports two incremental push modes: 1. Incremental push with auto-increment field check; 2. Incremental push of data with auxiliary tables and triggers.
[0055] like Figure 2As shown in the figure, when performing an incremental push with an auto-increment field check, the data that needs to be pushed includes auto-increment fields such as the auto-increment primary key / update time / creation time, and the auto-increment field needs to be specified in the configuration. After the first push, the largest auto-increment field of the currently pushed data will be recorded. After the push is successful, the next push will be checked and pushed based on this value, and the auto-increment field will be updated. If the push fails, the push will be retried. If the retries fail three times, it will end directly.
[0056] like Figure 3 As shown, when performing incremental data push with auxiliary tables and triggers, the data to be collected needs to have a primary key. And this program is allowed to create new tables and triggers in the source library. First check whether the incremental push task of the program's own service library has been executed. If not, perform a full push first, create a table of sync_<source table> in the source library, add a trigger to the source table, and configure the change type of incremental push: newly inserted data, newly updated data, and deleted data. If a full push has been executed, perform incremental push, obtain the primary key of the unsynchronized data according to sync_<source table>, find the data in the source table and synchronize it.
[0057] In addition, the data push end also supports pushing the same set of data to multiple destinations. When configuring push, it is allowed to initiate push to multiple destinations. It supports data signing and sends the signature to the data receiving end.
[0058] For scheduled push tasks, you can stop them. (For data integrity considerations, tasks in progress cannot be stopped.) Deleting a scheduled push task will simultaneously delete the trigger and auxiliary table created by the task. Therefore, when scheduled push is temporarily not needed, you can choose to stop the push. You should only delete the push task when you no longer need to change the scheduled push task.
[0059] S6: Get the pushed data through the data receiving end and perform data signature verification.
[0060] In a specific implementation manner, the push data and the key pair are obtained through the data receiving end; based on the key pair, the signature public key is imported, and the signature data string is verified based on the RSA or SM2 algorithm.
[0061] For example, the signature verification function needs to be enabled in the configuration. The signature verification is disabled by default. When it is enabled, you need to enter\import the signature public key (the signature public key is generated when the sender generates a signature key pair. If other ETL tools are used, you can generate the signature yourself. Signature and signature verification support algorithms such as RSA and SM2).
[0062] S7: After the data is verified, the pushed data is stored in the data receiving end.
[0063] In a specific implementation, the push data sent by the data push end is in json format. By default, data is stored in the form of the key of the incoming data corresponding to the field name of the selected data table. In the configuration, you can enable automatic conversion of camel case and underline naming formats, ignore case, and other configurations. In addition, you can also specify the field conversion of data in the configuration.
[0064] In addition, this method can also confirm the configuration information of the data push end and the data receiving end by generating test samples. Specifically, due to the use of API docking, the data receiving end or the data push end cannot obtain the table structure of the receiving and sending data source. Therefore, both the data receiving end and the data push end provide the function of generating test samples.
[0065] After completing the data collection and processing configuration and data signature configuration, the data push end can generate a test sample based on the current configuration to simulate the data style pushed during push.
[0066] After completing the data storage configuration, the data receiving end can also generate a test sample based on the current configuration (operations such as camel case conversion and ignoring case will not be reflected).
[0067] You can then compare the samples provided by both ends to determine the configuration of both ends. When generating test samples, data will not be actually extracted. All data is fake data generated based on field definitions.
[0068] In addition, in this method, both the data receiving end and the data pushing end support soft and hard processing when the configuration data does not exist.
[0069] The soft processing is: when the configured field does not exist, ignore the field, and record the key value as empty when extracting; ignore it directly when writing.
[0070] The hard processing is: when the configuration field does not exist, the program throws an exception and terminates the push or access.
[0071] The present invention discloses a data push method for non-confidential data. First, by finely configuring the data source, the connection between the data push end and the receiving end is ensured to be stable and reliable. In the data processing stage, the method can flexibly convert the information in the data source into a JSON string, which provides great convenience for the subsequent processing of the data. At the same time, by introducing a data signature and signature verification mechanism, the integrity and authenticity of the data during the transmission process are effectively guaranteed, and the risk of data tampering is prevented. In addition, the method also provides a variety of push processes to meet the data push requirements in different scenarios. In terms of configuration and testing, the method also performs well, ensuring the reliability and ease of use of the system. In summary, the method is a set of efficient, safe and reliable data push solutions with broad application prospects.
[0072] like Figure 4 As shown, the following is an embodiment of the data push system for non-confidential data provided by the embodiment of the present disclosure. The system and the data push method for non-confidential data in the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the data push system for non-confidential data, please refer to the embodiment of the data push method for non-confidential data mentioned above.
[0073] A data push system for non-confidential data includes: a data push terminal 1 and a data receiving terminal 2. The data push terminal 1 collects and pushes data and sends it to the data receiving terminal 2 through a network.
[0074] The data push terminal 1 supports basic data collection and push functions. It can extract the required data from the database according to the configuration and perform simple processing, and supports the generation of signatures for the data.
[0075] The data receiving end 2 supports visual parameter configuration, supports locking configuration, closing the visual configuration page, and can only be unlocked by manual intervention in the server, and other adjustable front-end configurations; supports data verification and data storage.
[0076] In addition, since the data receiving end 2 receives standard JSON input, it can also be used with the help of other ETL tools.
[0077] The data push terminal 1 includes: a push configuration module 101 , a data collection and processing module 102 , a data signature module 103 and a data push module 104 .
[0078] The push configuration module 101 is used to configure a data source for the data push terminal.
[0079] The data collection and processing module 102 is used to collect data from the data source, and generate a JSON string after data processing as the push data.
[0080] The data signature module 103 is used to sign the pushed data and configure the data receiving end.
[0081] The data push module 104 is used to execute the corresponding push process according to the push demand to send the push data to the data receiving end; the push process includes: a one-time push process, a scheduled push process, a full push process and an incremental push process.
[0082] The data receiving end 2 includes: a receiving configuration module 201 , a data acquisition and signature verification module 202 and a data storage module 203 .
[0083] The receiving configuration module 201 is used to configure a data source for a data receiving end and lock the configuration information.
[0084] The data acquisition and signature verification module 202 is used to acquire the pushed data and perform data signature verification.
[0085] The data storage module 203 is used to store the pushed data after the data signature is passed.
[0086] The data push system for non-confidential data provided in this embodiment realizes end-to-end scheduled and incremental push of non-confidential data, while ensuring the integrity and authenticity of the data during the data push process. This system is suitable for scenarios where the data receiving side does not allow direct exposure of the database. This system provides a plug-and-play data receiving end and provides data access support by exposing a limited API interface.
[0087] Figure 5 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.
[0088] The data push method for non-confidential data provided in the embodiment of the present application can be applied to electronic devices. It can be understood by those skilled in the art that the electronic device structure involved in the embodiment of the present invention does not constitute a limitation on the electronic device, and the electronic device may include more or less components than shown, or combine certain components, or arrange different components. In an embodiment of the present invention, the electronic device includes but is not limited to a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the embodiments of the present application described and / or required herein.
[0089] The electronic device may include a processor, an external memory interface, an internal memory, a universal serial bus (USB) interface, a charging management module, a power management module, a battery, a wireless communication module, an audio module, a speaker, a microphone, a sensor module, buttons, a camera, a display, and a SIM card interface, etc.
[0090] The processor may include one or more processing units, for example, the processor may include a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated into one or more processors.
[0091] The processor can be the nerve center and command center of the electronic device. The controller can generate an operation control signal according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.
[0092] A memory may also be provided in the processor for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. The memory may store instructions or data that the processor has just used or is cyclically used. If the processor needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor, and thus improves system efficiency.
[0093] The external memory interface can be used to connect an external memory card, such as a MicroSD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor through the external memory interface to implement data storage functions. For example, files such as music and videos can be saved in the external memory card.
[0094] The internal memory can be used to store computer executable program codes, which include instructions. The processor executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory. The internal memory may include a program storage area and a data storage area. The internal memory may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.
[0095] The wireless communication function of an electronic device can be realized through an antenna, a wireless communication module, a modem processor, and a baseband processor.
[0096] The wireless communication module can provide wireless communication solutions for electronic devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc.
[0097] Electronic devices can implement audio functions, etc. through audio modules, speakers, receivers, microphones, headphone jacks, and application processors.
[0098] Electronic devices can achieve shooting functions through ISP, camera, video codec, GPU, display and application processor.
[0099] Electronic devices can achieve display functions through GPU, display screen and application processor.
[0100] The GPU is a microprocessor for image processing that connects the display screen and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor may include one or more GPUs that execute program instructions to generate or change display information.
[0101] The display screen is used to display images, videos, etc. The display screen includes a display panel.
[0102] The above-mentioned electronic device implements the data push method for non-confidential data of the present application by finely configuring the data source, flexibly processing and generating JSON data, introducing data signature and verification mechanism, providing diversified push processes, and focusing on configuration flexibility and testing convenience, thereby achieving efficient, secure and reliable data transmission and reception, and achieving the beneficial effect of improving the flexibility of data push and the overall performance of the system.
[0103] The storage medium provided in the present application stores a program product that can implement a data push method for non-confidential data.
[0104] The data push method for non-confidential data includes: configuring a data source for a data push end; configuring a data source for a data receiving end, and locking the configuration information; collecting data from the data source through the data push end, generating a json string as push data after data processing; signing the push data through the data push end, and configuring the data receiving end; executing a corresponding push process according to the push demand to send the push data to the data receiving end; the push process includes: a one-time push process, a scheduled push process, a full push process, and an incremental push process; obtaining the push data through the data receiving end, and performing data signature verification; after the data signature verification is passed, the push data is put into storage through the data receiving end.
[0105] In some possible implementations, the data push method for non-confidential data disclosed herein can be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps described in the above "Exemplary Method" section of this specification according to various exemplary implementations of the present disclosure.
[0106] The storage medium of the present disclosure can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0107] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data push method for non-confidential data, characterized in that: include: Configure the data source for the data push end; Configure the data source for the data receiving end and lock the configuration information; The data push terminal collects data from the data source, processes the data, and generates a json string as the push data; Sign the pushed data through the data push end and configure the data receiving end; Execute the corresponding push process according to the push requirements to send the push data to the data receiving end; The push process includes: one-time push process, scheduled push process, full push process and incremental push process; Obtain the pushed data through the data receiving end and perform data signature verification; After the data is verified, the pushed data is stored in the data receiving end.
2. The method for pushing data for non-confidential data according to claim 1, characterized in that: The step of configuring a data source for the data push end includes: Preload the database JDBC package on the data push end and introduce it as an expansion of the data package when the program starts; Configure the database type, URL address, port, user name, password, database name, and mode name of the data source for the data push end; Reversibly encrypt and store passwords; The step of configuring a data source for the data receiving end and locking the configuration information includes: Configure a unique data push terminal for the data receiving terminal as the data source of the data receiving terminal; Configure related configuration items in the visual configuration interface of the data receiving end and lock the configuration information.
3. The data push method for non-confidential data according to claim 1, characterized in that: The data push end collects data from the data source, processes the data, and generates a json string as the push data, including: Connect to the database of the data source through the data push terminal, and collect field information in the specified table according to the pre-configured collection fields and field aliases; Perform value replacement and constant insertion on field information; Convert the processed field information into a JSON string as the push data.
4. The method for pushing data for non-confidential data according to claim 1, characterized in that: The data push end performs data signing on the pushed data and configures the data receiving end, including: Randomly generate an RSA key pair, generate a hash summary based on the pushed data, sign the hash summary with the private key, and generate a signature data string; Configure the address and port of the data receiving end for the data push end, and store the configuration information in the database.
5. The method for pushing data for non-confidential data according to claim 1, characterized in that: The push process includes: directly calling the push function to complete a data push for the push data; The timed push process includes: using multithreading and TimerTask to generate a timed push task, configuring the timed push information and recording the configuration information; when the timed push task is started, generating a corresponding working thread, and configuring the execution frequency through TimerTask; The full push process includes: checking the data volume of the push data, reading the push data in groups according to the data volume and the preset system configuration, and pushing the data to the data receiving end; The incremental push process adopts the auto-increment field check incremental push mode or the data incremental push mode with auxiliary tables and triggers; The auto-increment field check incremental push mode includes: specifying the auto-increment field in the push data; the auto-increment field includes the auto-increment primary key, update time or creation time; after completing the first push of the push data, recording the largest auto-increment field in the data pushed this time as a reference value, and checking the push based on the reference value when the next push of data starts; The data incremental push mode with auxiliary tables and triggers includes: Collect the primary key of the pushed data and allow the preset push program to create new tables and triggers in the source library; Check whether the incremental push task of the preset push program's own service library has been executed; If not executed, perform a full push first, create a source table in the source database, add a trigger to the source table, and configure the change type for incremental push; If full push has been executed, perform incremental push, obtain the primary key of the unsynchronized push data according to the source table, search for data in the source table and synchronize it.
6. The method for pushing data for non-confidential data according to claim 4, characterized in that: The step of obtaining the pushed data through the data receiving end and performing data signature verification includes: Obtain push data and key pair through the data receiving end; Based on the key pair, import the signature public key and verify the signature data string based on the RSA or SM2 algorithm.
7. The data push method for non-confidential data according to claim 1, characterized in that: The method further comprises: After the data push end completes the data signature of the pushed data, a first test sample is generated according to the current configuration information to simulate the data style of the pushed data; When the data receiving end completes the storage operation of pushing data, the second test sample is saved according to the current configuration information; The first test sample is compared with the second test sample, and configuration information of the data push end and the data receiving end is determined according to the comparison result.
8. A data push system for non-confidential data, characterized in that: The system adopts the data push method for non-confidential data as described in any one of claims 1 to 7; The system comprises: a data push end and a data receiving end, wherein the data push end collects and pushes data and sends the data to the data receiving end through a network; The data push end includes: Push configuration module, used to configure the data source for the data push end; The data collection and processing module is used to collect data from the data source, process the data and generate a json string as the push data; Data signature module, used to sign the pushed data and configure the data receiving end; The data push module is used to execute the corresponding push process according to the push demand to send the push data to the data receiving end; the push process includes: one-time push process, scheduled push process, full push process and incremental push process; The data receiving end comprises: A receiving configuration module is used to configure the data source for the data receiving end and lock the configuration information; Data acquisition and signature verification module, used to obtain pushed data and perform data signature verification; The data storage module is used to store the pushed data after the data verification is passed.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the data push method for non-confidential data as described in any one of claims 1 to 7 are implemented.
10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data push method for non-confidential data as described in any one of claims 1 to 7 are implemented.