Data processing method and device and computer storage medium

Abstract

The invention discloses a data processing method and device and a computer storage medium, and relates to the technical field of data processing.The method comprises the steps that service data sent by a slave node of a Redis platform is obtained, and after a master node of the Redis platform receives updating operation of the service data, the master node transmits the service data to the slave node of the Redis platform; when it is determined that the service data meets the encryption condition, the service data are sent to a database, and the database is used for encrypting the service data. According to the method and the device, the service data of the slave node is acquired by introducing an additional mechanism of a Redis observer, and the service data is sent to the database for national secret encryption, so that the confidentiality and the security of the service data are realized, the risk of data leakage and the risk of unauthorized access are effectively prevented, and the requirements of industries such as finance on data encryption are met.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device and computer storage medium. Background Art

[0002] In existing data storage systems, data persistence is a key technology to ensure that data is not lost. However, with the surge in data volume and the diversification of business needs, traditional data persistence methods have shown shortcomings in data security protection. Especially for the Redis platform, the inability to encrypt data significantly increases the risk of data leakage and unauthorized access. Summary of the invention

[0003] The main purpose of this application is to provide a data processing method, device and computer storage medium, aiming to solve the technical problem that the Redis platform cannot be encrypted and the data security is low.

[0004] To achieve the above purpose, an embodiment of the present application provides a data processing method, which is applied to Redis observer, and the method includes: Obtaining business data sent by a slave node of the Redis platform, wherein after the master node of the Redis platform receives an update operation of the business data, the master node transmits the business data to the slave node of the Redis platform; When it is determined that the business data meets the encryption condition, the business data is sent to a database, and the database is used to encrypt the business data.

[0005] In one embodiment, before the step of obtaining the service data sent by the slave node of the Redis platform, the following steps are included: Subscribe to the channel of the slave node of the Redis platform to receive update events; Based on the update event, identifying a service data update signal in the channel of the slave node; The service data is acquired from the slave node according to the update signal.

[0006] In one embodiment, after the step of acquiring the service data from the slave node according to the update signal, the following steps are included: Obtaining the business data and calculating a hash value of the business data; The hash value is compared with the hash value provided by the Redis slave node to perform integrity check on the business data.

[0007] In one embodiment, when it is determined that the business data meets the encryption condition, the business data is sent to a database, and the database is used to encrypt the business data, the step includes: Determining whether the business data meets predefined encryption conditions; When the business data meets the predefined encryption conditions, the business data is sent to the database and the business data is encrypted by an encryption algorithm; When the business data does not meet the predefined encryption conditions, the business data is preprocessed until the predefined encryption conditions are met, and then the business data is sent to the database for encryption operation.

[0008] In one embodiment, the step of determining whether the service data meets the predefined encryption condition includes: Extracting metadata information of the business data; Determining the sensitivity level of the business data according to the label or attribute of the metadata information; The sensitivity level of the business data is matched with a preset encryption rule, and it is determined whether to perform an encryption operation on the business data according to the matching result.

[0009] The present application provides a data processing method, which is applied to a node of a Redis platform. The method includes: Receive an instruction to update business data and determine the type of data operation that needs to be performed; Based on the data operation type, performing an update operation on the business data; When the business data is updated, an event notification mechanism is triggered to push the update event of the business data to the Redis observer.

[0010] The present application embodiment provides a data processing method, which is applied to Redis users, and the method includes: During program startup, the encrypted business data in the database is read, wherein the database engine automatically completes the decryption; Initialize the read business data to the Redis master node to ensure that the program can access the latest business data status when it starts.

[0011] In one embodiment, the step of initializing the read business data to the Redis master node includes: Obtain the decrypted business data, and serialize the decrypted business data according to the data type and structure supported by the Redis platform; The decrypted business data is sent to the master node, and the master node stores the decrypted business data after serialization processing to complete the initialization of the business data.

[0012] An embodiment of the present application also provides a data processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method described above.

[0013] An embodiment of the present application further provides a computer storage medium, which is a computer-readable storage medium. A computer program is stored on the computer storage medium. When the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0014] The embodiment of the present application discloses a data processing method, which obtains the business data sent by the slave node of the Redis platform, wherein after the master node of the Redis platform receives the update operation of the business data, the master node transmits the business data to the slave node of the Redis platform; when it is determined that the business data meets the encryption conditions, the business data is sent to the database, and the database is used to encrypt the business data. The present application obtains the business data of the slave node by introducing an additional mechanism of Redis observer, and sends the business data to the database for national secret encryption, thereby achieving the confidentiality and security of the business data, effectively preventing the risk of data leakage and the risk of unauthorized access, and meeting the requirements of the financial and other industries for data encryption. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flowchart of a first embodiment of a data processing method involved in an embodiment of the present application; Figure 2 This is a flow chart of a second embodiment of a data processing method according to an embodiment of the present application; Figure 3 This is a flow chart of a third embodiment of a data processing method according to an embodiment of the present application; Figure 4 A brief flowchart of a fourth embodiment of a data processing method according to an embodiment of the present application; Figure 5 This is a schematic diagram of the structure of the data processing equipment of this application.

[0016] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0017] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0018] In existing data storage systems, data persistence is a key technology to ensure that data is not lost. However, with the surge in data volume and the diversification of business needs, traditional data persistence methods have shown shortcomings in data security protection. Especially for the Redis platform, the inability to encrypt data significantly increases the risk of data leakage and unauthorized access.

[0019] Redis is an efficient in-memory data storage system suitable for scenarios that require high-speed reading and writing. In the financial industry, data security is of vital importance. The National Encryption Algorithm is an encryption algorithm of China's national standard, and the financial industry usually uses the National Encryption Algorithm to protect the security of data. However, Redis itself does not support National Encryption Algorithm encryption. In traditional usage scenarios, Redis is often used as a cache layer to improve application performance, while data persistence and encryption are performed by the backend database. Therefore, if you need to use Redis to write data from the node directly to the database and encrypt it with National Encryption, additional components or services are required to ensure that the data in Redis is encrypted with National Encryption when it is persisted to the database to meet data security requirements.

[0020] In order to solve the above-mentioned defects existing in the related art, the embodiment of the present application proposes a data processing method, which obtains the business data sent by the slave node of the Redis platform, wherein after the master node of the Redis platform receives the update operation of the business data, the master node transmits the business data to the slave node of the Redis platform; when it is determined that the business data meets the encryption conditions, the business data is sent to the database, and the database is used to encrypt the business data. This application obtains the business data of the slave node by introducing the additional mechanism of the Redis observer, and sends the business data to the database for national secret encryption, thereby achieving the confidentiality and security of the business data, effectively preventing the risk of data leakage and the risk of unauthorized access, and meeting the requirements of the financial and other industries for data encryption.

[0021] For ease of understanding, the technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0022] For the data processing method of the first embodiment proposed in this application, please refer to Figure 1 , applied to Redis observer, the method includes steps S10~S20: Step S10: Obtaining business data sent by the slave node of the Redis platform, wherein after the master node of the Redis platform receives the update operation of the business data, the master node transmits the business data to the slave node of the Redis platform.

[0023] It should be noted that the Redis platform is an open source, high-performance key-value database. Redis is famous for its extremely high read and write speeds and supports multiple types of data structures, such as strings, hashes, lists, sets, etc. Redis data is stored in memory, which enables it to provide fast access speeds and also supports data persistence, that is, saving the data in memory to disk periodically or as needed.

[0024] It should be noted that the master node refers to the main instance of the Redis platform. It serves as the data source in the master-slave replication mode and is responsible for maintaining the latest status of business data, processing write operation requests from various applications or clients, and synchronizing business data to slave nodes.

[0025] It should be noted that the slave node refers to the auxiliary instance of the Redis platform. The main function of the slave node is to receive and store the business data transmitted by the master node. In the master-slave replication mode, the slave node usually does not process write operation requests, but focuses on reading and backing up business data.

[0026] In an optional embodiment, step S10 may include: Step S110: subscribe to the channel of the slave node of the Redis platform to receive update events.

[0027] Optionally, after receiving the update event, the observer parses the update event to determine the data range or identifier that needs to be updated, including the key, type, and content of the business data change. Based on the parsed business data range or identifier, the observer then initiates a request to the slave node to pull the corresponding business data.

[0028] It should be noted that Redis observers include client applications, server logic modules, or other system components or services that have subscribed to update events. By subscribing to a channel, when a slave node publishes an update event on the channel, the observer can be notified of changes in business data in a timely manner and take corresponding actions based on the notification of the update event, such as updating the cache, triggering business logic, or logging.

[0029] Step S120: Based on the update event, identify the service data update signal in the channel of the slave node.

[0030] It should be noted that after receiving the update event, the business data update signal of the update event is parsed to extract the specific information of the business data update, including the key-value pair of the business data, the updated timestamp, the type of update operation (such as addition, deletion or modification), etc. According to the extracted business data update information, which business data has been updated is identified.

[0031] Step S130: Acquire the service data from the slave node according to the update signal.

[0032] It should be noted that after identifying the business data update signal, the specific business data to be obtained is determined according to the update signal. A request is sent to the Redis slave node to request the identified update data. After receiving the request, the slave node sends the corresponding business data to the observer, and the observer obtains the updated business data.

[0033] Optionally, after receiving the update signal issued by the Redis slave node and obtaining the business data from the Redis slave node, the business data is obtained and a hash calculation is performed to generate a hash value of the business data. The hash value is compared with the hash value provided by the slave node, and an integrity check of the business data is performed to verify the integrity of the data. If the hash value of the business data obtained by the observer matches the hash value provided by the slave node, the data integrity check passes, indicating that the data has not been tampered with during transmission. On the contrary, if the hash values ​​of the two do not match, the error handling mechanism is triggered, and the data needs to be re-acquired and / or the error information needs to be recorded.

[0034] For example, after receiving a batch of product data, the observer performs an integrity check on the product data, calculates a hash value corresponding to the product data, and compares it with the hash value provided by the slave node to ensure that the data has not been tampered with during transmission.

[0035] Optionally, a data integrity check is performed on the acquired business data to ensure that the received business data has not been tampered with during transmission and that the data content is complete and correct. In addition to calculating and comparing the hash value of the business data, the verification process can also determine whether the business data is complete by verifying the data signature or checking whether the data field complies with predefined rules.

[0036] It should be noted that data signature is a technology used to verify the source and integrity of data. The observer extracts the signature information from the business data and compares it with the stored signature information. If the signature information matches, it indicates that the source of the business data is reliable and has not been tampered with.

[0037] It should be noted that checking whether the data fields comply with the predefined rules includes verifying the data type, data format, data range, etc. The observer checks whether the data fields meet the requirements according to the preset rule set. If all fields comply with the rules, it means that the business data content is complete and correct.

[0038] Step S20: When it is determined that the business data meets the encryption condition, the business data is sent to a database, and the database is used to encrypt the business data.

[0039] Optionally, when writing business data to the database, use the national secret algorithm to encrypt the business data in the database to ensure that the data is strongly encrypted when stored. Select a suitable national secret algorithm, such as SM2 for public key encryption, SM4 for symmetric encryption, and SM3 for hash algorithm. Then generate the key required for encryption according to the national secret algorithm, and use the key to encrypt the business data. The encrypted business data only exists in ciphertext during storage.

[0040] After completing the encryption of business data, follow the database storage rules and write the encrypted business data into the database to ensure that the data is securely stored in the database according to the predetermined structure. At the same time, record the business data storage operation log, which contains information such as the time of the storage operation, the content of the stored data, the encryption algorithm and key used. By recording the business data storage operation log, each step of the data processing can be traced, and when data security is threatened, the problem can be quickly located and measures can be taken.

[0041] It should be noted that the national secret algorithm is a set of encryption algorithm standards issued by the China National Cryptography Administration, including SM1, SM2, SM3, SM4 and other algorithms, which are widely used in finance, government and important information systems.

[0042] In an optional embodiment, step S20 may include: Step S210: Determine whether the business data meets the predefined encryption conditions.

[0043] Optionally, metadata information of the business data is extracted. Based on the labels or attributes of the metadata information, the sensitivity level of the business data is determined. For example, if the business data contains personal identity information or financial data, it may be marked as a high sensitivity level. The sensitivity level of the business data is matched with the preset encryption rules, and it is determined whether to perform encryption operations on the business data based on the matching results. The encryption rules define how business data of different sensitivity levels should be encrypted, or if encryption is required in certain circumstances.

[0044] It should be noted that metadata information refers to data information that describes business data, including key information such as business data attributes, tags, sources, creation time, etc.

[0045] Step S220: When the business data meets the predefined encryption condition, the business data is sent to the database, and the business data is encrypted using an encryption algorithm.

[0046] In an optional embodiment, the business data is evaluated to determine whether the business data meets the predefined encryption conditions. When the business data meets the predefined encryption conditions, the business data is directly sent to the database and encrypted using an encryption algorithm.

[0047] It should be noted that the predefined encryption conditions include the sensitivity level of business data, specific data labels or metadata attributes, and business rules or legal and regulatory requirements. The data format conversion process ensures that the business data is consistent with the database storage specifications in terms of structure and encoding, which may include structural adjustments, encoding conversions, or necessary data cleaning.

[0048] It is understood that the encryption operation of the database involves the use of security keys, which can be generated by the database management system or provided by the key management service to ensure the security of the encryption process. After the encryption of the business data is completed, the business data is securely stored and can be accessed and used when needed through the query and decryption functions of the database.

[0049] Optionally, in order to ensure the security of business data, the database will encrypt the business data when writing it to the database. The encryption process includes selecting an appropriate national encryption algorithm, managing encryption keys, etc. After the business data is encrypted, the encrypted business data will be persisted to the disk, so that even if the database is subject to unauthorized access, the business data will not be easily leaked.

[0050] Step S230: When the business data does not meet the predefined encryption conditions, the business data is preprocessed until the predefined encryption conditions are met, and the business data is sent to the database for encryption.

[0051] If the business data does not meet the predefined encryption conditions, the following operations will be performed: First, the business data is preprocessed to adapt to the storage requirements of the database, including structural adjustment, encoding conversion, data cleaning, mapping field names, reorganizing data relationships, or applying transformation logic. Then, the preprocessed data is sent to the database. In the database, even if the business data is not identified as a high sensitivity level, it will be encrypted using an encryption algorithm, using a key generated by the database management system or provided by the key management service. The encrypted data will be stored on disk to ensure storage security. At the same time, key log information of the preprocessing and encryption process is recorded.

[0052] Exemplarily, after obtaining a batch of product data and confirming that the product data is complete and correct, the observer pre-processes the product data and converts it into the format requirements required by the database, such as mapping the product name to the field name in the database, adjusting the data type of the product price, etc. After completing the format conversion of the business data, the product data can be written to the database. The database first generates the key required for SM4 encryption, and then uses the key to encrypt the product data. After encryption is completed, the product data follows the storage rules of the database to ensure that the product data is securely stored according to the predetermined structure. In the process of writing product data to the database, the observer records the data storage operation log, including the time of the storage operation, the content of the stored data, the encryption algorithm and key used, and other information.

[0053] In this embodiment, the Redis observer, as an intermediate processing device between the Redis platform and the database, undertakes the important tasks of encrypting and securely transmitting business data. After obtaining the business data, the Redis observer performs the following processing flow: First, the Redis observer initializes the key management system, generates or imports key pairs based on national encryption standards, and securely stores private keys, while appropriately disclosing or distributing public keys to relevant entities. Next, configure encryption parameters, select appropriate encryption algorithms, such as SM4 symmetric encryption or SM2 asymmetric encryption, and set appropriate key lengths and encryption modes. In addition, the Redis observer also defines a set of security policies, which include using the SM3 hash algorithm for data integrity verification, and setting the encryption levels of the transport layer and storage layer according to business needs.

[0054] In the data receiving and preprocessing stage, after the Redis observer obtains the business data, it parses the business data and identifies the data segments that need to be encrypted. In order to improve encryption efficiency, the Redis observer can perform necessary preprocessing on the business data, including format conversion and data compression, to reduce the amount of data in the encryption process.

[0055] In the encryption processing phase, the Redis observer can adopt a hybrid encryption mode according to business needs. First, a session key is negotiated between the sender (Redis slave node) and the receiver (database) through the SM2 algorithm. Subsequently, the business data is encrypted in groups using the SM4 symmetric encryption algorithm. At the same time, the hash value of the business data is calculated and attached to the encrypted data to verify the integrity of the data.

[0056] During the transmission and storage phase, the Redis observer encapsulates the encrypted business data and hash values ​​into a standard data packet format and transmits them to the target database or storage medium through a secure channel.

[0057] In the database decryption and verification phase, the encrypted data can be decrypted using the corresponding national encryption algorithm and session key, and the integrity of the business data can be verified through hash calculation to ensure that the data has not been tampered with during transmission. Finally, the decrypted business data is post-processed, such as format recovery and data decompression.

[0058] During the entire encryption process, the Redis observer performs security auditing and monitoring operations, recording key information during the encryption and decryption process, including timestamps, operation status, key usage, etc. At the same time, the Redis observer monitors the operation status in real time. If any abnormal behavior is detected, an alarm will be issued in time when a problem is found to ensure the stable operation of the system.

[0059] In order to maintain long-term data security, Redis observers regularly update key pairs during key update and rotation phases to ensure key security. During the update process, a smooth transition between old and new keys is achieved to ensure business continuity and business data security.

[0060] The data processing method of this embodiment provides a data security mechanism. By introducing additional components or services (Redis observers), the business data of the Redis slave nodes is monitored and synchronized in real time, ensuring the consistency and real-time performance of the data within the Redis platform. At the same time, an encryption condition judgment mechanism is introduced so that only business data that meets specific security requirements will be transmitted to the database and encrypted using the national encryption algorithm, thereby enhancing the confidentiality and integrity of the data. In this way, even if the Redis slave node does not directly process the write operation request, it can ensure that the data is encrypted when it is written to the database, thereby meeting the high requirements of the financial industry for data security.

[0061] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiment can refer to the above introduction, and will not be repeated later. Figure 2 The data processing method of the embodiment of the present application is applied to the node of the Redis platform, and the method includes steps S40 to S60: Step S40: receiving an instruction to update the business data and determining the type of data operation to be performed.

[0062] Optionally, the application receives an update instruction for the business data, wherein the update instruction may be sent through a user interface input, an API (Application Programming Interface) call, or other communication methods. The application generates a corresponding update request based on the received update instruction, and the update request includes the type of data operation to be performed, such as adding, modifying, or deleting data.

[0063] Exemplarily, the client sends an update instruction through an API call, and the update instruction includes a specific business data identifier and the type of operation to be performed. After the master node receives the update request containing the update instruction, it extracts the business data identifier, the operation type, and the necessary data fields from the update request. The master node maps the corresponding operation type to the data processing logic inside the master node. For example, if the operation type is "add", the data insertion process is ready to be executed; if the operation type is "modify", the data update process is ready to be executed; if the operation instruction is "delete", the data removal process is ready to be executed.

[0064] Optionally, after receiving the update request corresponding to the update instruction, the master node first starts the built-in verification mechanism to verify the validity of the update request, including checking the format of the update request, operation authority and data validity. After the verification is passed, the master node will execute the update operation in the update request and perform corresponding modification operations on the records of existing business data, including adding, modifying or deleting existing business data records.

[0065] It should be noted that during the format verification phase of the update request, the built-in verification mechanism of the master node will check whether the format of the update request complies with the predetermined protocol specifications, such as whether it contains the necessary operation type, business data identifier, and the business data itself that needs to be updated.

[0066] Specifically, after receiving the update request, the master node first parses the header structure and content structure of the update request to ensure that the update request follows the correct data exchange format, such as JSON (JavaScript Object Notation), XML (eXtensible Markup Language) or other custom formats. Then compare whether the field name, data type, and field order in the update request match the preset template, and whether there are any errors or exceptions that do not conform to the data exchange format. If the format verification of the update request fails, the master node will refuse to perform the corresponding update operation and return format error information to the requester.

[0067] It should be noted that the verification phase of the operation authority needs to ensure that the client or application sending the update request has the authority to execute the update request operation.

[0068] Specifically, the master node verifies the user identity information contained in the update request, such as API keys, digital signatures, or access tokens, and compares the user identity information with the authorization information stored in the user permission database. The permission verification stage can finely control access rights based on user roles, data categories, and specific operations, and check whether the requester has the right to perform the type of operation in the update request, such as write permissions or the right to modify specific data. If the permission verification fails, the master node will reject the update request.

[0069] It should be noted that data validity verification is to ensure that the submitted data meets the constraints of business logic.

[0070] Specifically, the master node checks data fields according to pre-configured rule sets, such as data types, data dependencies, data value ranges, and the presence of required fields, to ensure that all data meets business requirements. For example, whether numeric fields are within the allowed range, whether string fields conform to the specified format, whether dates are valid dates, and whether required fields are correctly filled in.

[0071] Step S50: Based on the data operation type, an update operation is performed on the business data.

[0072] Optionally, after identifying the data operation type of the update request, the master node performs a corresponding update operation on the business data according to the data operation type, including adding new business data, updating existing business data, or deleting business data. After completing the update operation, the master node sends a confirmation message to the requester (i.e., the application), wherein the confirmation message includes feedback on the successful execution of the update operation or problems encountered during the execution process.

[0073] Exemplarily, the master node executes a built-in verification mechanism to check whether the operation permissions of the update instruction comply with the preset security policy, and confirm the validity of the business data, such as whether the data format is correct, whether the data value is within the allowed range, etc. After the verification process is correct, the master node performs corresponding operations on the business data according to the operation type. For example, when the operation type is an "add" operation, the master node will create a new record in the database and fill the extracted data field into the corresponding field; when the operation type is a "modify" operation, the master node will find the corresponding data record based on the business data identifier and update the relevant fields; when the operation type is a "delete" operation, the master node will locate the specified record and execute the delete command.

[0074] After all update operations are completed, the master node sends a confirmation message to the client through the communication interface, which contains the status code of the operation result and necessary description information. If the operation is successful, the status code will indicate success and contain the newly created data identifier or the updated data snapshot; if the operation fails, the status code will indicate an error with a description of the error cause.

[0075] Optionally, the master node first identifies the operation type in the update request. The operation type can be to add, modify or delete a business data record, and each operation type corresponds to a specific data processing step.

[0076] In the add operation, the master node inserts the new business data record into the business data set, ensuring that the new business data is compatible with the existing business data structure. In the modify operation, the master node locates the existing business data record and changes the field value according to the information in the update request, while ensuring the consistency of data dependencies and business rules. The delete operation involves removing business data records that are no longer needed or do not meet the retention conditions from the business data set.

[0077] When executing data operation types, the master node maintains the consistency and integrity of business data, handles the associations and dependencies between business data, and performs transactional operations to ensure the consistency of business data status.

[0078] It should be noted that after the master node completes the business data update operation, the master node will start the synchronization mechanism to transmit the updated business data to at least one slave node of the Redis platform to ensure the consistency of the business data on all nodes.

[0079] In an optional embodiment, step S50 includes: Step S310: Obtain the replication offset of the slave node, and determine the difference value between the replication offsets of the slave node and the master node.

[0080] Optionally, a stable communication link is established between the master node and the slave node through a socket connection to determine the transmission path for synchronization of business data. The master node executes a psync command to determine the data difference value between the slave node and the master node's replication offset based on its own replication offset and the slave node's replication offset. Based on the response of the psync command, it is determined whether the difference value between the slave node's replication offset and the master node's replication offset exceeds a preset offset threshold.

[0081] It should be noted that a socket connection is a communication path between two network devices (usually computers). This connection is usually based on the TCP / IP protocol (Transmission Control Protocol / Internet Protocol) to ensure reliable transmission of data in the network. When two devices need to communicate, they exchange information through a socket connection. In Redis's master-slave replication, a stable communication link is established between the master node and the slave node through a socket connection, allowing the master node to send data to the slave node, and the slave node to receive the data and apply the data to its own database.

[0082] Optionally, the master node sends a psync command to the slave node, carrying its own replication offset information. After receiving the psync command, the slave node compares its own replication offset with the master node's replication offset and calculates the difference between the slave node and the master node. Subsequently, the slave node updates its own replication offset based on the response to the psync command.

[0083] It should be noted that the psync command is a command used for data synchronization in the master-slave replication environment of the Redis database. The psync command determines whether to perform partial replication or full replication by comparing the replication offsets of the master and slave nodes. Partial replication can reduce the consumption of network bandwidth and master node resources, while full replication is used to resynchronize the slave node after it has been disconnected for a long time. The psync command ensures data consistency between the master and slave nodes and is a key component of the Redis master-slave replication environment.

[0084] Step S320: When the difference value exceeds a preset offset threshold, full replication is performed to generate a data snapshot and transmit it to the slave node.

[0085] Step S330: When the difference value does not exceed the preset offset threshold, partial replication is performed to transmit the updated part of the business data to the slave node.

[0086] Optionally, when the difference value exceeds the preset offset threshold, the master node will execute the full replication process, generate an RDB (Redis Database Backup) snapshot of the current business data state, and send the RDB snapshot to the slave node through the socket connection. The slave node can use the RDB snapshot to rebuild the business data state consistent with the master node.

[0087] When the difference value does not exceed the preset offset threshold, the master node only needs to perform partial replication, by copying the business data in the buffer, constructing the difference part of the business data missing from the slave node, and sending the difference data to the slave node.

[0088] After completing the data synchronization operation, the slave node will trigger an update event to notify the relevant subscribers or services of the business data change information, ensuring that the entire system can respond to changes in business data in a timely manner.

[0089] For example, the replication offset of the master node is set to 10000, and the replication offset of the slave node is set to 9500. The master node sends a psync command to the slave node, and attaches the replication offset information of the master node itself. After receiving the psync command, the slave node compares its own replication offset with the replication offset of the master node, and calculates the difference value to be 500.

[0090] According to the response of the psync command, if the difference value of 500 does not exceed the preset offset threshold, the master node instructs partial replication. The slave node then receives the difference data sent by the master node, which contains all data changes between the replication offset of 9500 and 10000. The slave node applies this difference data to its own database and updates its own replication offset to 10000 to keep it consistent with the master node.

[0091] Step S60: When the business data is updated, an event notification mechanism is triggered to push the business data update event to the Redis observer.

[0092] Optionally, the slave node enables the publish-subscribe function and sets the monitoring conditions to identify the business data updates of the master node. When the slave node receives the updated business data, it triggers the event notification mechanism and pushes the business data update event to the Redis observer. The update event contains detailed information about the business data change, including the change type, the timestamp of the change, and the affected data range. After the slave node generates an update event, it sends the update event to the observer.

[0093] It should be noted that the publish-subscribe function is a messaging mechanism of the Redis platform that allows different parts of the system to communicate through message publishing and subscription. Publishers send messages to specific channels, and subscribers listen to these channels to receive messages. By subscribing to channels, slave nodes can receive all data update events published by the master node. When the master node completes the update operation of business data and decides to notify the slave nodes of the changes in these business data, the mechanism will encapsulate the update event into a message and publish it to the corresponding channel. After subscribing to the channel, the slave node will continue to listen to these channels. Once a new message is published, the slave node will immediately receive the update event and identify and process it according to the preset listening conditions. The processing process includes synchronous updates to local data storage, triggering other business processes, or notifying other components in the system.

[0094] Optionally, after receiving the business data, the slave node synchronizes the business data to the cache to ensure that the business data stored in the slave node remains consistent with the master node, while also providing the slave node itself with data backup and fast access capabilities. When the slave node completes the synchronization of the business data, it generates an update event containing the updated content of the business data. The slave node publishes the update event to the channel subscribed by the Redis observer through the publish-subscribe function. All Redis observers subscribed to the channel, including client applications, server-side logic modules, or other system components, can receive the update notification. After receiving the update event, the observer can perform corresponding operations based on the content of the update event, such as refreshing the local cache, updating the data displayed in the user interface, or triggering the background data processing process.

[0095] Exemplarily, after receiving the business data from the node, the event notification mechanism is triggered to generate an update event. The notification content of the update event is as follows: "Business data change notification: the change type is 'update', the change timestamp is '2023-07-2112:34:56', and the affected data range is offset 9500 to 10000". After the slave node updates the event, the update event is notified to the observers, including the client that subscribed to the update event and the server responsible for caching the update.

[0096] After the observer receives the notification of the update event, the client will re-query the latest business data to update the user interface, and the server will update the cache according to the content of the update event to ensure that the business data in the cache is consistent with the business data in the database. In addition, the server may also trigger related business logic, such as performing data analysis and report generation tasks. Through the above event notification mechanism, the components in the entire system can respond quickly and coordinately to changes in business data.

[0097] This embodiment receives business data update instructions and performs data operations on the Redis master node, and at the same time, the built-in verification mechanism ensures the legitimacy and data accuracy of the update request, effectively preventing invalid or malicious operations. The psync command is used to manage data synchronization between the master and slave nodes, and the full or partial replication strategy is flexibly selected according to the difference value, which not only optimizes the system resource consumption, but also significantly improves the fault tolerance and high availability of the system, and is particularly suitable for industries such as finance that have extremely high requirements for data security and business continuity. In addition, the publish-subscribe function is enabled on the Redis slave node and the monitoring conditions are set to achieve instant response and synchronization to the business data update of the master node, ensuring the real-time and consistency of the data. After the business data synchronization is completed, the update event is pushed to the Redis observer by triggering the event notification mechanism, which promotes the circulation and sharing of information within the system, ensuring that the system can quickly respond to business data changes and execute subsequent business logic, thereby comprehensively improving the data processing flexibility and response speed of the system.

[0098] Based on the above embodiment, in the third embodiment of the present application, the same or similar contents as those in the above embodiment can be referred to the above introduction, and will not be described in detail later. Figure 3 The data processing method of the embodiment of the present application is applied to Redis users, and the method may also include: Step S80: During program startup, the encrypted business data in the database is read, wherein the database engine automatically completes the decryption.

[0099] In an optional implementation scheme, before the application or client sends a request to update business data to the master node, it must first ensure that the master node has the latest business data. First, the application needs to read the existing encrypted business data from the database, and trigger the database engine to retrieve the encrypted business data from the database through the query request instruction. The database engine locates the required business data record according to the query conditions contained in the instruction, and automatically decrypts the encrypted business data using the stored security key and decryption algorithm to obtain the original business data. After the decryption operation is completed, the database engine transmits the original business data to the application.

[0100] It should be noted that during the Redis application startup process, the encrypted business data is read from the database and automatically decrypted, ensuring that the security and integrity of the business data are guaranteed even when the master node fails or goes down. The automatic decryption function of the database engine not only improves the efficiency of data recovery, but also enhances the security of data during the recovery process and prevents the risk of data leakage during transmission. By quickly restoring business data to the Redis master node, the reliability of the system and business continuity are improved, ensuring that the application can quickly resume normal operation when the master node is unavailable, effectively avoiding business interruptions caused by data loss.

[0101] Step S90: Initialize the read business data to the Redis master node to ensure that the program can access the latest business data status when it starts.

[0102] It should be noted that when the application starts, the business data of the master node needs to be initialized to ensure that the master node has accurate, complete and up-to-date business data status when the application starts, so that the application can start processing business logic immediately without waiting for business data to be synchronized from other data sources.

[0103] In an optional implementation, step S90 includes: Step S410: Acquire the decrypted business data, and serialize the decrypted business data according to the data type and structure supported by the Redis platform.

[0104] Optionally, obtain the decrypted business data. The decrypted business data may be stored in a persistent database or other data storage medium and retrieved when the application starts. After obtaining the business data, convert the business data into a data type and structure suitable for the Redis platform. Serialization ensures that the business data format matches the requirements of Redis, including but not limited to Redis native data types such as strings, lists, sets, ordered sets, and hashes.

[0105] During the serialization process, the application may need to perform operations such as data structure conversion, encoding format adaptation, and data field mapping. For example, if the original data is stored in a relational database, the application will need to convert it into a key-value pair format for storage in Redis. In addition, serialization may also involve data compression, chunking, or other optimization measures to improve data transmission efficiency and storage performance.

[0106] Step S420: Send the decrypted business data to the master node, and the master node stores the decrypted business data after serialization processing to complete the initialization of the business data.

[0107] It should be noted that Redis application refers to software applications or services that interact with the Redis database. Redis is responsible for performing various operations on the Redis database, including reading encrypted data, decrypting, serializing data to adapt to Redis data structures, and initializing or updating data to the Redis master node. The Redis application ensures that the master node can quickly access the latest business data when the system starts, and processes business data update requests from users or other services to maintain the real-time and consistency of data.

[0108] Optionally, after serialization is completed, the application sends the serialized business data to the Redis master node and stores the data in the master node by executing corresponding Redis commands, such as SET, HSET, or LPUSH. When the original business data is successfully initialized to the master node to ensure that the master node has the latest business data status, the application can send a business data update request to the master node.

[0109] It should be noted that the master node receives the original business data sent by the application through the Redis command, determines the storage method and location of the data, and performs corresponding data storage operations based on the command content, including adding the original business data to a list, set, ordered set or hash table, or updating an existing key-value pair.

[0110] This embodiment ensures that the application can quickly access the latest business data by reading and decrypting the business data in the database during the Redis application startup process. The serialization process adapts the decrypted data to the data types and structures supported by Redis, improving the compatibility and transmission efficiency of the data with the Redis platform. The serialized data is sent to the Redis master node and initialized, providing an accurate, complete and up-to-date data foundation for the application.

[0111] In the fourth embodiment of the present application, the same or similar contents as those in the above embodiment can be referred to the above description, and no further description will be given later. Figure 4 , Figure 4 A brief flow chart of a data processing method is provided, specifically: In the Redis application program, data initialization is required first. Data initialization includes reading encrypted business data from the database, and initializing the business data to the Redis master node after the database engine automatically completes decryption. When the Redis application updates data, it sends an update operation to the Redis master node, where the update operation can be to add, modify, or delete data. Once the data of the Redis master node changes, the Redis synchronization mechanism will synchronize the changed data to the Redis slave node. After the Redis slave node receives the business data change, the Redis update event notification mechanism will be triggered, which is responsible for sending the update event notification to the Redis observer. After receiving the notification, the Redis observer will pull the updated business data from the Redis slave node. Finally, the Redis observer writes the obtained business data to the persistent database. In the database, the business data will be encrypted with national secrets, and the encrypted business data will be persisted to the disk to complete the persistent encrypted storage of the business data.

[0112] This embodiment effectively ensures the consistency and real-time update of business data by implementing data initialization and synchronization update processes in the Redis application program. Automatic database decryption ensures the security of data during the initialization phase, while Redis' synchronization mechanism and event notification mechanism ensure that data changes can be quickly and accurately propagated within the system. The observer writes the updated data into the database and performs national encryption, further enhancing the security of the data and meeting the business needs of high security standards.

[0113] The present application provides a data processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data processing method in the above-mentioned embodiment.

[0114] Reference below Figure 5 , which shows a schematic diagram of the structure of a data processing device suitable for implementing the embodiment of the present application. The data processing device in the embodiment of the present application may include but is not limited to network element devices such as switches, routers, etc. Figure 5 The data processing device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0115] like Figure 5 As shown, the data processing device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data processing device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data processing device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a data processing device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0116] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0117] The data processing device provided by the present application adopts the data processing device method in the above embodiment, which can solve the technical problem that the Redis platform cannot be encrypted and the data security is low. Compared with the prior art, the beneficial effects of the data processing device provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and the other technical features in the data processing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0118] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0119] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0120] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the data processing method in the above-mentioned embodiment.

[0121] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0122] The computer-readable storage medium may be included in a data processing device, or may exist independently without being incorporated into a data processing device.

[0123] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by a data processing device, the data processing device: obtains business data sent by a slave node of the Redis platform, wherein after the master node of the Redis platform receives an update operation of the business data, the master node transmits the business data to the slave node of the Redis platform; when it is determined that the business data meets the encryption condition, the business data is sent to a database, and the database is used to encrypt the business data.

[0124] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0125] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0126] The modules involved in the embodiments described in the present application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0127] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data processing method, and can solve the technical problem that the Redis platform cannot be encrypted and the data security is low. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the data processing method provided in the above-mentioned embodiment, and will not be repeated here.

[0128] An embodiment of the present application provides a computer program product, including a computer program, which implements the steps of the above-mentioned data processing method when executed by a processor.

[0129] The computer program product provided in this application can solve the technical problem that the Redis platform cannot be encrypted and the data security is low. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiment of this application are the same as the beneficial effects of the data processing method provided in the above embodiment, which will not be repeated here.

[0130] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

[0131] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0132] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0133] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data processing method, characterized in that: Applied to Redis observer, the method includes: Obtaining business data sent by a slave node of the Redis platform, wherein after the master node of the Redis platform receives an update operation of the business data, the master node transmits the business data to the slave node of the Redis platform; When it is determined that the business data meets the encryption condition, the business data is sent to a database, and the database is used to encrypt the business data.

2. The method according to claim 1, characterized in that Before the step of obtaining the business data sent by the slave node of the Redis platform, the method includes: Subscribe to the channel of the slave node of the Redis platform to receive update events; Based on the update event, identifying a service data update signal in the channel of the slave node; The service data is acquired from the slave node according to the update signal.

3. The method according to claim 2, characterized in that After the step of acquiring the service data from the slave node according to the update signal, the method further comprises: Obtaining the business data and calculating a hash value of the business data; The hash value is compared with the hash value provided by the Redis slave node to perform integrity check on the business data.

4. The method according to claim 1, characterized in that When it is determined that the business data meets the encryption condition, the business data is sent to a database, and the database is used to encrypt the business data, the step comprising: Determining whether the business data meets predefined encryption conditions; When the business data meets the predefined encryption conditions, the business data is sent to the database and the business data is encrypted by an encryption algorithm; When the business data does not meet the predefined encryption conditions, the business data is preprocessed until the predefined encryption conditions are met, and then the business data is sent to the database for encryption operation.

5. The method according to claim 4, characterized in that The step of determining whether the service data meets the predefined encryption condition comprises: Extracting metadata information of the business data; Determining the sensitivity level of the business data according to the label or attribute of the metadata information; The sensitivity level of the business data is matched with a preset encryption rule, and it is determined whether to perform an encryption operation on the business data according to the matching result.

6. A data processing method, characterized in that: Applied to a node of a Redis platform, the method comprises: Receive an instruction to update business data and determine the type of data operation that needs to be performed; Based on the data operation type, performing an update operation on the business data; When the business data is updated, an event notification mechanism is triggered to push the update event of the business data to the Redis observer.

7. A data processing method, characterized in that: Applied to Redis users, the method includes: During program startup, the encrypted business data in the database is read, wherein the database engine automatically completes the decryption; Initialize the read business data to the Redis master node to ensure that the program can access the latest business data status when it starts.

8. The method according to claim 7, characterized in that The step of initializing the read business data to the Redis master node includes: Obtain the decrypted business data, and serialize the decrypted business data according to the data type and structure supported by the Redis platform; The decrypted business data is sent to the master node, and the master node stores the decrypted business data after serialization processing to complete the initialization of the business data.

9. A data processing device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method according to any one of claims 1 to 8.

10. A computer storage medium, characterized in that: The computer storage medium is a computer-readable storage medium, and a computer program is stored on the computer storage medium. When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 8 are implemented.

Images