A data processing method and device, computer equipment and storage medium

Abstract

The application discloses a data processing method and device, a computer device and a storage medium, and relates to the technical field of data processing. The method comprises the following steps: acquiring a first data flow set; determining a service identifier set based on the first data flow set; changing a first access address corresponding to each service identifier in the service identifier set into a second access address when a source database and a target database perform data synchronization; and changing the second access address into a target database after the source database and the target database complete data synchronization. Through the above method, the data access address is changed into the target database after the source database and the target database complete data synchronization, so that the migration of service access can be completed on the basis of completing data synchronization, thereby avoiding the problem of service access address failure in the data synchronization process, avoiding the problem of service interruption, and ensuring the service quality of the service when data synchronization is performed.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a data processing method and apparatus, computer equipment and storage medium. Background Technology

[0002] With the development of internet services and the increasing demands of mobile terminals, the amount of data generated in daily life is growing rapidly. Different databases have different functional characteristics, thus requiring the use of different databases for different aspects of data management to achieve efficient data management. This makes data synchronization between different databases increasingly important. Currently, when synchronizing data providing online services, the typical process involves first copying all data of the object to be migrated from the source database to the destination database. Then, the access route for the object to be migrated is switched from the source database to the destination database so that after data synchronization is complete, access requests to the target object are routed to the destination database for processing. However, during data synchronization, if an access request to the target object is received, this request may still be routed to the source database. In this case, the source database will reject the access request, potentially leading to business interruption or the risk of double-writing during data synchronization, thereby reducing the quality of service. Therefore, ensuring the quality of service during data synchronization is a pressing issue that needs to be addressed. Summary of the Invention

[0003] This application provides a data processing method, apparatus, computer device, and storage medium. When synchronizing data between a source database and a target database, the data access address is changed from the source database to the target database, but the data access address still points to the source database. After the source database and the target database complete data synchronization, the data access address is changed to point to the target database. This allows for the migration of business access while completing data synchronization, thereby avoiding the problem of business access address failure during data synchronization and preventing business interruption. This ensures the quality of service during data synchronization.

[0004] In view of the above, the first aspect of this application provides a data processing method, comprising:

[0005] Obtain a first data traffic set, wherein the first data traffic set includes first data traffic generated by multiple services accessing a first access address within a first time period, the first access address belongs to the source database and points to the source database, and each first data traffic carries a service identifier, the service identifier indicating the service;

[0006] A service identifier set is determined based on a first data traffic set, wherein the service identifier set includes multiple service identifiers, each service identifier indicating a service that accesses a first access address within a first time period, and each service identifier corresponds to a first access address.

[0007] When synchronizing data between the source database and the target database, the first access address corresponding to each business identifier in the business identifier set is changed to the second access address, wherein the second access address belongs to the target database and points to the source database.

[0008] After the source database and the target database have completed data synchronization, the second access address is changed to point to the target database.

[0009] A second aspect of this application provides a data processing apparatus, comprising:

[0010] The acquisition module is used to acquire a first data traffic set, wherein the first data traffic set includes first data traffic generated by multiple services accessing a first access address within a first time period, the first access address belongs to the source database and points to the source database, and each first data traffic carries a service identifier, the service identifier indicating the service.

[0011] The determination module is used to determine a service identifier set based on a first data traffic set, wherein the service identifier set includes multiple service identifiers, each service identifier indicating a service that accesses a first access address within a first time period, and each service identifier corresponds to a first access address;

[0012] The update module is used to change the first access address corresponding to each business identifier in the business identifier set to the second access address when synchronizing data between the source database and the target database. The second access address belongs to the target database and points to the source database.

[0013] The update module is also used to change the second access address to point to the target database after the source database and target database have completed data synchronization.

[0014] In one possible implementation, the data processing apparatus further includes a setting module;

[0015] The setting module is used to set a first parameter on the source database after the update module changes the first access address corresponding to each service identifier in the service identifier set to the second access address, and before the update module changes the second access address to point to the target database. The first parameter instructs the source database to stop writing operations and instructs the source database to disconnect the data connection with the service indicated by the service identifier in the service identifier set.

[0016] The update module is specifically used to set a second parameter on the source database after the source database and the target database have completed data synchronization. The second parameter instructs the source database to stop read / write operations.

[0017] If the data connection between the source database and the business indicated by the business identifier in the business identifier set are both preset values, then the second access address will be changed to point to the target database.

[0018] In one possible implementation, a setting module is specifically used to obtain a second data traffic set, wherein the second data traffic set includes second data traffic generated by multiple services accessing a second access address within a second time period, each second data traffic carrying a service identifier, and the second time period being the time period for data synchronization between the source database and the target database;

[0019] If the second data traffic in the second data traffic set carries multiple service identifiers, then it is determined that the first access address has been completely switched to the second access address;

[0020] If data synchronization proceeds normally, set the first parameter on the source database.

[0021] In one possible implementation, the data processing apparatus further includes a synchronization module;

[0022] The synchronization module is used to obtain backup data from the source database and import the backup data into the target database;

[0023] Incremental data from the source database is synchronized to the target database in real time to complete the data synchronization between the source and target databases.

[0024] In one possible implementation, the acquisition module is specifically used to issue a data traffic acquisition instruction, wherein the data traffic acquisition instruction instructs the source database to acquire the data traffic that accesses the first access address within a first time period;

[0025] The data traffic obtained from the received data traffic acquisition command is used as the first data traffic set.

[0026] In one possible implementation, each data traffic in the data traffic set includes the service origin address;

[0027] The determination module is specifically used to determine the access curve of accessing the first access address within the first time period based on the access source address included in each data traffic in the data traffic set.

[0028] Based on the access curve of accessing the first access address within the first time period, multiple services that access the first access address within the first time period are identified to obtain a set of service identifiers.

[0029] A third aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods described in the above aspects.

[0030] A fourth aspect of this application provides a computer device, characterized in that it includes: a memory, a transceiver, a processor, and a bus system; wherein the memory is used to store a program; the processor is used to execute the program in the memory to implement the methods described in the above aspects; and the bus system is used to connect the memory and the processor to enable the memory and the processor to communicate.

[0031] A fifth aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods described in the above aspects.

[0032] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0033] This application provides a data processing method. First, a first data traffic set is obtained. This first data traffic set includes first data traffic generated by multiple services accessing a first access address within a first time period. The first access address belongs to a source database and points to the source database. Each first data traffic carries a service identifier, which indicates a service. Then, a service identifier set is determined based on the first data traffic set. This service identifier set includes multiple service identifiers, each indicating a service accessing the first access address within the first time period. Each service identifier corresponds to a first access address. Based on this, during data synchronization between the source database and the target database, the first access address corresponding to each service identifier in the service identifier set is changed to a second access address. The service identifier set includes multiple service identifiers, each indicating a service, and each service identifier corresponds to a second access address. The second access address belongs to the target database and points to the source database. After data synchronization between the source database and the target database is completed, the second access address is changed to point to the target database. Using the above method, when synchronizing data between the source and target databases, the data access address is changed from the source database to the target database, but the data access address still points to the source database. After the source and target databases have completed data synchronization, the data access address is changed to point to the target database. This allows for the migration of business access while completing data synchronization, thus avoiding the problem of business access address failure during data synchronization and preventing business interruption. This ensures the quality of service during data synchronization. Attached Figure Description

[0034] Figure 1 This is a system schematic diagram of a data processing method in an embodiment of this application;

[0035] Figure 2 A flowchart illustrating the data processing method provided in this application embodiment;

[0036] Figure 3 A schematic diagram of an embodiment of the data processing method provided in this application;

[0037] Figure 4 A schematic diagram of one embodiment of the first data traffic set provided in this application;

[0038] Figure 5 A schematic diagram of one embodiment of the switching access address provided in this application;

[0039] Figure 6A A schematic diagram illustrating one embodiment of service switching provided in this application;

[0040] Figure 6BA schematic diagram illustrating another embodiment of service switching provided in this application;

[0041] Figure 6C A schematic diagram illustrating another embodiment of service switching provided in this application;

[0042] Figure 7 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0043] Figure 8 This is a schematic diagram of the server structure in an embodiment of this application. Detailed Implementation

[0044] This application provides a data processing method, apparatus, computer device, and storage medium. When synchronizing data between a source database and a target database, the data access address is changed from the source database to the target database, but the data access address still points to the source database. After the source database and the target database complete data synchronization, the data access address is changed to point to the target database. This allows for the migration of business access while completing data synchronization, thereby avoiding the problem of business access address failure during data synchronization and preventing business interruption. This ensures the quality of service during data synchronization.

[0045] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0046] With the development of internet services and the increasing demands of mobile terminals, the amount of data generated in daily life is growing rapidly. Different databases have different functional characteristics, thus requiring the use of different databases for different aspects of data management to achieve efficient data management. This makes data synchronization between different databases increasingly important. Currently, when synchronizing data providing online services, the typical process involves first copying all data of the object to be migrated from the source database to the destination database. Then, the access route for the object to be migrated is switched from the source database to the destination database so that after data synchronization is complete, access requests to the target object are routed to the destination database for processing. However, during data synchronization, if an access request to the target object is received, this request may still be routed to the source database. In this case, the source database will reject the access request, potentially leading to business interruption or the risk of double-writing during data synchronization, thereby reducing the quality of service. Therefore, ensuring the quality of service during data synchronization is a pressing issue that needs to be addressed. Based on this, the embodiments of this application provide a data processing method that can not only complete data synchronization but also migrate business access, avoid the problem of business access address failure during data synchronization, and avoid business interruption, thereby ensuring the quality of service during data synchronization.

[0047] First, for ease of understanding, some terms or concepts involved in the embodiments of this application will be explained.

[0048] I. Data Transformation Service (DTS):

[0049] DTS is a real-time data synchronization tool for database migration in a relational database management system (MySQL).

[0050] II. Network Data Acquisition Tool (TCPDUMP)

[0051] TcpDump is a network data collection and analysis tool. Specifically, it is a packet analysis tool that captures data packets on a specified machine and analyzes them according to user-defined criteria.

[0052] III. Terminate the connection

[0053] Kill connection means terminating the data connection on the database.

[0054] The application system of this application embodiment is described below. Please refer to [link / reference]. Figure 1 , Figure 1This is a system schematic diagram of a data processing method in an embodiment of this application, such as... Figure 1 As shown, this video processing system includes a source database, a target database, and a service platform. All three are deployed on a server, and the data processing method is specifically executed by the service platform. Based on this, the service platform obtains multiple first data traffic flows accessing a first access address within a first time period. This first access address belongs to the source database. Then, based on the data traffic set, multiple service identifiers are determined. Therefore, during data synchronization between the source and target databases, the service platform changes the first access address corresponding to each service identifier in the service identifier set to a second access address. Each service identifier corresponds to a second access address, which belongs to the target database and points to the source database. This avoids the problem of service access address failure during data synchronization, thus preventing service interruption. Furthermore, after the source and target databases complete data synchronization, the second access address is changed to point to the target database. In addition to completing data synchronization, the migration of service access can also be completed, further ensuring the quality of service during data synchronization.

[0055] It should be noted that, Figure 1 The servers described herein can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Servers can be directly or indirectly connected via wireless networks, wired networks, or removable storage media. The aforementioned wireless networks use standard communication technologies and / or protocols. Wireless networks are typically the Internet, but can also be any network, including but not limited to Bluetooth, Local Area Network (LAN), Metropolitan Area Network (MAN), Wide Area Network (WAN), mobile, private networks, or any combination of virtual private networks. In some embodiments, custom or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies. Removable storage media can be Universal Serial Bus (USB) flash drives, external hard drives, or other removable storage media.

[0056] Secondly, although Figure 1 Only one source database, one target database, and one service platform are shown, but it should be understood that... Figure 1The examples provided are for understanding this solution only. The specific number of source databases, target databases, and service platforms should be flexibly determined based on the actual situation.

[0057] Secondly, the embodiments of this invention can be applied to various scenarios, including but not limited to cloud technology, intelligent transportation, and assisted driving. Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology applied based on the cloud computing business model. It can form resource pools, be used on demand, and is flexible and convenient. Cloud computing technology will become an important support. The backend services of technical network systems require a large amount of computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the Internet industry, every item may have its own identification mark in the future, all of which need to be transmitted to the backend system for logical processing. Data of different levels will be processed separately, and various industry data will all require strong system support, which can only be achieved through cloud computing.

[0058] Cloud computing refers to the delivery and usage model of IT infrastructure, meaning obtaining necessary resources through a network in an on-demand and easily scalable manner. In a broader sense, cloud computing refers to the delivery and usage model of services, meaning obtaining necessary services through a network in an on-demand and easily scalable manner. These services can be IT and software related, internet-related, or other services. Cloud computing is a product of the development and integration of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing.

[0059] With the development of the internet, real-time data streams, and the diversification of connected devices, as well as the demands for search services, social networks, mobile commerce, and open collaboration, cloud computing has developed rapidly. Unlike previous parallel distributed computing, cloud computing will revolutionize the entire internet model and enterprise management model from a conceptual standpoint.

[0060] Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of storage devices of various types (storage devices are also called storage nodes) in the network to work together and jointly provide data storage and business access functions to the outside world.

[0061] Currently, the storage method of storage systems is as follows: Logical volumes are created. During the creation of a logical volume, physical storage space is allocated to each logical volume. This physical storage space may consist of a single storage device or the disks of several storage devices. Clients store data on a logical volume, which means storing the data on the file system. The file system divides the data into many parts, each part being an object. Each object contains not only the data but also additional information such as a data identifier (ID, ID entity). The file system writes each object to the physical storage space of that logical volume and records the storage location information of each object. Therefore, when a client requests access to data, the file system can allow the client to access the data based on the storage location information of each object.

[0062] The process by which a storage system allocates physical storage space to a logical volume is as follows: the physical storage space is pre-divided into strips according to the capacity estimate of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of Redundant Array of Independent Disks (RAID). A logical volume can be understood as a strip, thus allocating physical storage space to the logical volume.

[0063] For easier understanding, please refer to Figure 2 , Figure 2 Please refer to the flowchart illustrating the data processing method provided in this application embodiment. Figure 2 The data processing method consists of three stages: data traffic capture and analysis, data synchronization, and access address binding and switching. The functions and processes of each stage will be described below:

[0064] In the data traffic capture and analysis phase A1, the data flow capture program specifically includes a server and a client. The client of the data flow capture program encapsulates the TCPDUMP network capture command and sends the encapsulated network capture command to the server where the source database is located for real-time packet capture. The client of the data flow capture program captures the first data traffic generated by multiple services accessing the first access address within the first time period. At this time, the first access address belongs to the source database and points to the source database. Then, the client of the data flow capture program uploads the first data traffic generated by multiple services accessing the first access address within the first time period to the server of the data flow capture program (this server also belongs to the service platform described in this embodiment). Based on this, the service platform can obtain the first data traffic set, which includes the first data traffic generated by multiple services accessing the first access address within the first time period. Each first data traffic carries a service identifier, which indicates the service. Therefore, it can be determined which services are accessing the source database within the first time period. Based on this, the service platform determines a set of service identifiers based on the service identifier carried by each first data traffic in the first data traffic set. That is, the set of service identifiers includes multiple service identifiers, and each service identifier indicates a service accessing a first access address within a first time period. Each service identifier corresponds to a first access address. It should be understood that in practical applications, the first time period in this solution can be a real-time dynamic time period, and this is not limited here.

[0065] In data synchronization phase A2, the service platform configures the source database to synchronize data to the target database in real time. The Data Transformation Service (DTS) provides the real-time database data synchronization technology for this solution. The data processing method provided in this embodiment relies on this real-time data synchronization technology to ensure real-time data synchronization between the source and target databases. The DTS data processing method requires obtaining backup data from the source database and importing the backup data into the target database. After importation, incremental data from the source database is synchronized to the target database in real time using master-slave replication technology to complete the data synchronization between the source and target databases.

[0066] In the access address binding switching phase A3, which is specifically implemented during the data synchronization phase A2, that is, when the source database and the target database are synchronizing data, the service platform binds the routing of the target database's access address (i.e., the second access address) to the source database. Specifically, it changes the first access address corresponding to each business identifier in the business identifier set to the second access address. At this point, the second access address belongs to the target database and points to the source database. This ensures that regardless of whether the first access address (belonging to the source database) or the second access address (belonging to the target database) is used to access the data, the actual data source accessed is always the source database.

[0067] Furthermore, the service platform will gradually switch the configuration of each first data traffic in the first data traffic set obtained in the data traffic capture and analysis phase A1 to the second access address until all access sources are switched to the second access address. At this point, each data traffic in the data traffic set obtained in real-time packet capture will access the second access address. To avoid the problem of double writing of data, after all accesses are switched to the second access address, the service platform will check whether the data synchronization is normal. If it is normal, the service platform will add a global read lock on the source data and set a first parameter on the source database. This first parameter instructs the source database to stop write operations and disconnect the data connection with the business indicated by the business identifier in the business identifier set. Specifically, this means setting a read-only parameter on the source database, which is the first parameter, thereby locking the write operations of the source database and terminating the connection (Killconnection), that is, terminating all connections on the source database except for the data synchronization task.

[0068] Based on this, after the service platform completes data synchronization between the source and target databases, it sets a second parameter on the source database. This second parameter instructs the source database to stop read / write operations. At this point, when the data connections between the source database and the services indicated by the service identifiers in the service identifier set are both preset values ​​(0 in this embodiment), the second access address is changed to point to the target database, that is, the routing of the second access address is redirected back to the target database. After successfully redirecting the second access address to the target database, all service connections will be migrated to the target database. Thus, while maintaining the service quality during data synchronization, the efficiency of data migration is also guaranteed.

[0069] Based on the above introduction, the data processing method in this application will be described below, taking the terminal device as the executing entity as an example. Please refer to [link / reference]. Figure 3 , Figure 3 A schematic diagram of one embodiment of the data processing method provided in this application is shown below. Figure 3 As shown, the method includes:

[0070] 101. Obtain the first data traffic set.

[0071] In this embodiment, the service platform obtains the first data traffic generated by multiple services accessing the first access address within a first time period, thereby obtaining a first data traffic set including the aforementioned multiple first data traffic. The aforementioned first access address belongs to the source database and points to the source database. Each first data traffic carries a service identifier, which indicates the service. Therefore, it can be known which services are accessing the source database within the first time period.

[0072] For ease of understanding, Figure 4 A schematic diagram of one embodiment of the first data traffic set provided in this application is shown, such as... Figure 4 As shown, B1 refers to the first data traffic 1, B2 refers to the first data traffic 2, and B3 refers to the first data traffic 3. If the first data traffic 1 carries service identifier 1, the first data traffic 2 carries service identifier 2, and the first data traffic 3 carries service identifier 3, and service identifier 1 indicates service 1, service identifier 2 indicates service 2, and service identifier 3 indicates service 3, then based on... Figure 4 It can be seen that services 1, 2, and 13 all access the first access address within the first time period. The data traffic generated by service 1 accessing the first access address within the first time period is designated as first data traffic B1, the data traffic generated by service 2 accessing the first access address within the first time period is designated as first data traffic B2, and the data traffic generated by service 1 accessing the first access address within the first time period is designated as first data traffic B3. It should be understood that... Figure 4 The examples provided are for understanding this solution only. In practical applications, the first time period in this solution can be a real-time dynamic time period, which is not limited here.

[0073] 102. Determine the service identifier set based on the first data traffic set.

[0074] In this embodiment, since each first data traffic in the first data traffic set carries a service identifier in step 101, the service platform can obtain a service identifier set by performing data analysis on the first data traffic set. This service identifier set includes multiple service identifiers, and each service identifier indicates a service accessing the first access address within a first time period. Each service identifier corresponds to the first access address. For example, the first data traffic set includes first data traffic 1, first data traffic 2, and first data traffic 3, and first data traffic 1 carries service identifier 1, first data traffic 2 carries service identifier 2, and first data traffic 3 carries service identifier 3. Therefore, the service platform can obtain a service identifier set by performing data analysis on the first data traffic set, and the service identifier set includes service identifier 1, service identifier 2, and service identifier 3. Each service identifier indicates a service accessing the first access address within the first time period.

[0075] 103. When synchronizing data between the source database and the target database, change the first access address corresponding to each business identifier in the business identifier set to the second access address.

[0076] In this embodiment, the service platform also includes a presentation layer connected to a terminal device with a display interface. Therefore, during data synchronization between the source database and the target database, a data selection interface needs to be displayed on the terminal device connected to the presentation layer within the service platform. Based on user input regarding which access address to migrate the service to, and after completing the preset permission password verification corresponding to the second access address of the target database, the first access address corresponding to each service identifier in the service identifier set is changed to the second access address. This means the routing of the second access address belonging to the target data source is bound to the source database. At this point, the second access address belongs to the target database and points to the source database. This ensures that regardless of whether the first access address belonging to the source database or the second access address belonging to the target database is used to access the data, the actual data source accessed is always the source database, thus avoiding interruption of user access to service data and guaranteeing the quality of service. Specifically, the data synchronization described in this embodiment is a mechanism used to maintain data content consistency between two different databases. The target database is the database that needs to maintain data content consistency with the source database. Therefore, the source database is the database that needs to synchronize local data to the target database, and the target database is the database that needs to obtain data from the source database in real time.

[0077] For ease of understanding, Figure 5 A schematic diagram of one embodiment of the switching access address provided in this application is shown below. Figure 5As shown, C1 refers to the second access address belonging to the target database, and C2 refers to the preset permission password corresponding to the second access address. Based on this, users can select the second access address C1 belonging to the target database according to their needs, and operate on it, entering the preset permission password C2 corresponding to the second access address to complete password verification. This confirms that the business accessing the source database needs to be switched to the second access address C2 belonging to the target database. Therefore, after the data synchronization between the source and target databases is completed, users can also access the corresponding business data by accessing the second access address C2 belonging to the target database. It should be understood that... Figure 5 This is only to help users understand how to select the second access address to be switched through the terminal device's display interface. The specific access address and corresponding permission password should be flexibly determined according to actual needs.

[0078] 104. After the source database and the target database have completed data synchronization, change the second access address to point to the target database.

[0079] In this embodiment, after the service platform completes data synchronization between the source database and the target database, it changes the second access address to point to the target database. Specifically, after completing data synchronization between the source and target databases, the second access address is changed to point to the target database. This means that when accessing business data based on the second access address, the actual data source accessed is changed to the target database. In other words, it is no longer necessary to obtain the required business data by accessing the source database. Thus, while completing data synchronization, business data migration is achieved, and business interruption is avoided.

[0080] In this embodiment, a data processing method is provided. Using the above method, when synchronizing data between the source database and the target database, the data access address is changed from the source database to the target database, but the data access address still points to the source database. After the source database and the target database complete data synchronization, the data access address is changed to point to the target database. Thus, on the basis of completing data synchronization, the migration of business access can also be completed, thereby avoiding the problem of business access address failure during data synchronization, thus avoiding the problem of business interruption, and ensuring the service quality of business during data synchronization.

[0081] Optionally, in the above Figure 3 Based on the corresponding embodiments, in an optional embodiment of the data processing method provided in this application, after changing the first access address corresponding to each service identifier in the service identifier set to a second access address, and before changing the second access address to point to the target database, the data processing method further includes:

[0082] Set a first parameter on the source database, wherein the first parameter instructs the source database to stop writing operations and instructs the source database to disconnect the data connection with the service indicated by the service identifier in the service identifier set;

[0083] After the source and target databases have completed data synchronization, the second access address will be changed to point to the target database. Specifically, this includes:

[0084] After the source database and the target database have completed data synchronization, a second parameter is set on the source database, which instructs the source database to stop read / write operations;

[0085] If the data connection between the source database and the business indicated by the business identifier in the business identifier set are both preset values, then the second access address will be changed to point to the target database.

[0086] In this embodiment, to avoid the problem of double writing, after changing the first access address corresponding to each business identifier in the business identifier set to the second access address, and before changing the second access address to point to the target database, the service platform will also check whether the data synchronization between the source database and the target database is normal. If the data synchronization between the source database and the target database is normal, the service platform will add a global read lock on the source database and set a first parameter on the source database. This first parameter instructs the source database to stop write operations and disconnect the data connection with the business indicated by the business identifier in the business identifier set. Specifically, the read_only parameter is set on the source database, which is the first parameter, thereby locking the write operation of the source database and killing the connection, that is, terminating all connections on the source database except for the data synchronization task. Therefore, the source database can disconnect the data connection with the business indicated by the business identifier in the business identifier set, preventing the business from directly accessing the first access address of the original database, thereby avoiding the problem of double writing.

[0087] Based on this, the service platform also needs to lock and modify the first parameter (read_only parameter). After the source database completes the aforementioned Kill connection operation, if the service platform determines that the source database and the target database have completed data synchronization, the service platform also needs to set the second parameter on the source database. The second parameter instructs the source database to stop read / write operations, that is, to lock the read / write operations of the source database, so as to prevent the business from reading the corresponding business data from the source database when accessing the second access address of the target database. After completing this Kill connection operation, it checks whether the data connection between the source database and the business indicated by the business identifier in the business identifier set is all the preset value (0 in this embodiment), that is, to determine that all connections are disconnected. Only then will the service platform change the second access address to point to the target database, that is, to bind the route of the second access address back to the target database. Then all subsequent business connection access requests will access the second access address, and based on the second access address, business data will be directly obtained from the target database.

[0088] For ease of understanding, Figures 6A to 6C A schematic diagram illustrating an embodiment of service switching provided in this application is shown below. Figure 6A As shown, a communication connection exists between the source database and the target database. Since the routing for the second access address is bound to the source database, if a service simultaneously sends a data request carrying the first access address to the source database and a data request carrying the second access address to the target database, it will cause a double-write problem for the corresponding service data. Therefore, the service platform needs to set a first parameter, which instructs the source database to stop write operations and disconnect its data connection with either service. Thus, as... Figure 6B As shown, the service cannot send a service data request carrying the first access address to the source database. Instead, it can only send a service data request carrying the second access address to the target database. The target database obtains the corresponding service data from the source database through the routing of the second access address, and then sends the service data to complete the service for that service.

[0089] Therefore, after confirming that the source and target databases have completed data synchronization, the service platform still needs to set a second parameter on the source database. This second parameter instructs the source database to stop read / write operations, effectively locking it and preventing services from accessing the target database's second access address by retrieving corresponding business data from the source database. After completing this "Kill connection" operation, the service platform checks that the source database and the business are completely disconnected. Only then will the service platform change the second access address to point to the target database, thus obtaining the desired result. Figure 6CThe service platform will redirect the routing of the second access address back to the target database. All services that have successfully switched will only be able to send business data requests carrying the second access address to the target database. Furthermore, since the source and target databases have already synchronized, the target database will directly retrieve business data from the target database based on the second access address. It should be understood that... Figures 6A to 6C This is only for understanding how to switch specific business processes. The specific business processes and parameter settings should be flexibly determined based on business access requirements and scenarios.

[0090] This embodiment provides another data processing method. Using the method described above, by setting a first parameter on the source database, the problem of double writing to the same business data during data synchronization is avoided. Secondly, after data synchronization is complete, by setting a second parameter on the source database again, it is ensured that there is no further interaction of business data between the source and target databases. This ensures that all subsequent business access requests will be switched to the target database, thereby guaranteeing the reliability of data migration.

[0091] Optionally, in the above Figure 3 Based on the corresponding embodiments, in an optional embodiment of the data processing method provided in this application, setting a first parameter on the source database specifically includes:

[0092] Obtain a second set of data traffic, wherein the second set of data traffic includes second data traffic generated by multiple services accessing the second access address within a second time period, each second data traffic carries a service identifier, and the second time period is the time period for data synchronization between the source database and the target database;

[0093] If the second data traffic in the second data traffic set carries multiple service identifiers, then it is determined that the first access address has been completely switched to the second access address;

[0094] If data synchronization proceeds normally, set the first parameter on the source database.

[0095] In this embodiment, when setting the first parameter, the service platform needs to obtain the second data traffic set. The second data traffic set includes the second data traffic generated by multiple services accessing the second access address within the second time period. Each second data traffic carries a service identifier. The second time period is the time period for data synchronization between the source database and the target database, which means that the second time period is later than the first time period.

[0096] Based on this, the service platform then gradually switches the configuration of each second data traffic in the acquired second data traffic set to the second access address until all access sources have switched to the second access address. In other words, during data synchronization, each second data traffic in the second data traffic set acquired through real-time packet capture accesses the second access address. Based on this, the service platform also needs to determine whether, within the time period of data synchronization between the source and target databases, the second data traffic accessed by the business in the second data traffic set carries multiple business identifiers, thus confirming that the first access address has been completely switched to the second access address. Then, it checks whether the data synchronization between the source and target databases is normal. If the data synchronization between the source and target databases is normal, the service platform will add a global read lock on the source data and set the first parameter on the source database.

[0097] This embodiment provides another data processing method. Using the method described above, all traffic data generated by real-time switching of service access is switched to the target database, ensuring that the real-time access address has been successfully changed. Furthermore, after all services that previously accessed the source database access the target database, the system checks whether data synchronization is running normally before setting the first parameter on the source database. This ensures normal read and write operations for non-dual-write service data before the source database disconnects the service connection, guaranteeing the reliability of data read and write processing and further improving the reliability of data processing in this solution.

[0098] Optionally, in the above Figure 3 Based on the corresponding embodiments, in one optional embodiment of the data processing method provided in this application, the source database and the target database are synchronized, specifically including:

[0099] Retrieve backup data from the source database and import the backup data into the target database;

[0100] Incremental data from the source database is synchronized to the target database in real time to complete the data synchronization between the source and target databases.

[0101] In this embodiment, the service platform is configured to synchronize data from the source database to the target database in real time. The Data Transformation Service (DTS) provides the real-time database data synchronization technology, and the data processing method provided in this embodiment relies on this real-time data synchronization technology to ensure real-time data synchronization between the source and target databases. Based on this, the service platform needs to obtain backup data from the source database and import the backup data into the target database. After the import is complete, incremental data from the source database is synchronized to the target database in real time using master-slave replication technology to complete the data synchronization between the source and target databases. This ensures the reliability of data synchronization in this embodiment and guarantees the real-time nature of data synchronization even when incremental data appears.

[0102] Optionally, in the above Figure 3 Based on the corresponding embodiments, in one optional embodiment of the data processing method provided in this application, obtaining a data traffic set specifically includes:

[0103] A data traffic acquisition instruction is issued, which instructs the source database to acquire the data traffic that accesses the first access address within the first time period.

[0104] The data traffic obtained from the received data traffic acquisition command is used as the first data traffic set.

[0105] In this embodiment, the service platform specifically obtains the first data traffic set through a data flow packet capture program. The data flow packet capture program includes a server (i.e., the service platform described in this embodiment) and a client. The client of the data flow packet capture program encapsulates a TCPDUMP network packet capture command and sends the encapsulated command to the server containing the source database for real-time packet capture. At this time, the first access address belongs to the source database and points to it. Then, the client of the data flow packet capture program uploads the first data traffic generated by multiple services accessing the first access address within a first time period to the service platform, thus obtaining the data traffic that the service platform receives as the first data traffic set. Therefore, the obtained first data traffic set includes the first data traffic generated by multiple services accessing the first access address within a first time period, and each first data traffic carries a service identifier, which indicates the service. Therefore, it is possible to know which services are accessing the source database within the first time period.

[0106] Optionally, in the above Figure 3 Based on the corresponding embodiments, in an optional embodiment of the data processing method provided in this application, each data traffic in the data traffic set includes a service source address;

[0107] The service identifier set is determined based on the first data traffic set, specifically including:

[0108] Based on the access source address included in each data traffic in the data traffic set, determine the access curve for accessing the first access address within the first time period;

[0109] Based on the access curve of accessing the first access address within the first time period, multiple services that access the first access address within the first time period are identified to obtain a set of service identifiers.

[0110] In this embodiment, each data traffic in the data traffic set includes a service source address. Based on this, the service platform can determine the access curve for accessing the first access address within the first time period based on the access source address included in each data traffic in the data traffic set. (See also...) Figure 4 , Figure 4 The curve corresponding to each service is the access curve corresponding to the traffic generated by that service accessing the first access address within the first time period. This allows the service platform to determine the multiple services accessing the first access address within the first time period based on the access curves. (See also...) Figure 4 It can be seen that the first data traffic B1, the first data traffic B2 and the first data traffic B3 correspond to service 1, service 2 and service 3 respectively. Therefore, it can be determined that the multiple services that access the first access address within the first time period include service 1, service 2 and service 3. Thus, a service identifier set is obtained, and the service identifier set includes service identifier 1 (indicating service 1), service identifier 2 (indicating service 2) and service identifier 3 (indicating service 3).

[0111] Figure 7 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application, as shown below. Figure 7 As shown, the data processing device 700 includes:

[0112] The acquisition module 701 is used to acquire a first data traffic set, wherein the first data traffic set includes first data traffic generated by multiple services accessing a first access address within a first time period, the first access address belongs to the source database and points to the source database, and each first data traffic carries a service identifier, the service identifier indicating the service.

[0113] The determination module 702 is used to determine a service identifier set based on a first data traffic set, wherein the service identifier set includes multiple service identifiers, each service identifier indicating a service that accesses a first access address within a first time period, and each service identifier corresponds to a first access address;

[0114] The update module 703 is used to change the first access address corresponding to each business identifier in the business identifier set to the second access address when synchronizing data between the source database and the target database. The second access address belongs to the target database and points to the source database.

[0115] The update module 703 is also used to change the second access address to point to the target database after the source database and the target database have completed data synchronization.

[0116] Optionally, in the above Figure 7 Based on the corresponding embodiments, in another embodiment of the data processing apparatus provided in this application, the data processing apparatus 700 further includes a setting module 704;

[0117] The setting module 704 is used to set a first parameter on the source database after the update module 703 changes the first access address corresponding to each service identifier in the service identifier set to the second access address, and before the update module 703 changes the second access address to point to the target database. The first parameter instructs the source database to stop writing operations and instructs the source database to disconnect the data connection with the service indicated by the service identifier in the service identifier set.

[0118] The update module 703 is specifically used to set a second parameter on the source database after the source database and the target database have completed data synchronization. The second parameter instructs the source database to stop read / write operations.

[0119] If the data connection between the source database and the business indicated by the business identifier in the business identifier set are both preset values, then the second access address will be changed to point to the target database.

[0120] Optionally, in the above Figure 7 Based on the corresponding embodiments, in another embodiment of the data processing device provided in this application, the setting module 704 is specifically used to obtain a second data traffic set, wherein the second data traffic set includes second data traffic generated by multiple services accessing the second access address within a second time period, each second data traffic carries a service identifier, and the second time period is the time period for data synchronization between the source database and the target database;

[0121] If the second data traffic in the second data traffic set carries multiple service identifiers, then it is determined that the first access address has been completely switched to the second access address;

[0122] If data synchronization proceeds normally, set the first parameter on the source database.

[0123] Optionally, in the above Figure 7Based on the corresponding embodiments, in another embodiment of the data processing apparatus provided in this application, the data processing apparatus 700 further includes a synchronization module 705;

[0124] Synchronization module 705 is used to obtain backup data based on the source database and import the backup data into the target database;

[0125] Incremental data from the source database is synchronized to the target database in real time to complete the data synchronization between the source and target databases.

[0126] Optionally, in the above Figure 7 Based on the corresponding embodiments, in another embodiment of the data processing device provided in this application, the acquisition module 701 is specifically used to issue a data traffic acquisition instruction, wherein the data traffic acquisition instruction instructs the source database to acquire the data traffic accessing the first access address within a first time period;

[0127] The data traffic obtained from the received data traffic acquisition command is used as the first data traffic set.

[0128] Optionally, in the above Figure 7 Based on the corresponding embodiments, in another embodiment of the data processing apparatus provided in this application, each data traffic in the data traffic set includes a service source address;

[0129] The determination module 702 is specifically used to determine the access curve of accessing the first access address within the first time period based on the access source address included in each data traffic in the data traffic set;

[0130] Based on the access curve of accessing the first access address within the first time period, multiple services that access the first access address within the first time period are identified to obtain a set of service identifiers.

[0131] This application also provides another data processing device, all of which are deployed on a server. Please refer to [link to relevant documentation]. Figure 8 , Figure 8This is a schematic diagram of one embodiment of the server in this application. As shown in the figure, the server 1000 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1022 (e.g., one or more processors) and memory 1032, and one or more storage media 1030 (e.g., one or more mass storage devices) for storing application programs 1042 or data 1044. The memory 1032 and storage media 1030 can be temporary or persistent storage. The program stored in the storage media 1030 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the server. Furthermore, the CPU 1022 may be configured to communicate with the storage media 1030 and execute the series of instruction operations in the storage media 1030 on the server 1000.

[0132] Server 1000 may also include one or more power supplies 1026, one or more wired or wireless network interfaces 1050, one or more input / output interfaces 1058, and / or one or more operating systems 1041, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0133] The steps performed by the server in the above embodiments can be based on this Figure 8 The server structure shown.

[0134] The server includes 1022 CPUs used for performing tasks such as... Figure 3 The illustrated embodiments and Figure 3 The corresponding embodiments.

[0135] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the aforementioned actions. Figure 3 The steps performed by the service platform in the method described in the illustrated embodiment.

[0136] This application also provides a computer program product including a program, which, when run on a computer, causes the computer to perform the aforementioned actions. Figure 3 The steps performed by the service platform in the method described in the illustrated embodiment.

[0137] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0138] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, at least two units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0139] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across at least two network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0140] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0141] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0142] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, include: Obtain a first data traffic set, wherein the first data traffic set includes first data traffic generated by multiple services accessing a first access address within a first time period, the first access address belongs to a source database and points to the source database, and each first data traffic carries a service identifier, the service identifier indicating the service; A service identifier set is determined based on the first data traffic set, wherein the service identifier set includes multiple service identifiers, each service identifier indicating a service that accesses the first access address within the first time period, and each service identifier corresponds to the first access address; When the source database and the target database are synchronizing data, the first access address corresponding to each service identifier in the service identifier set is changed to a second access address, wherein the second access address belongs to the target database and points to the source database; Obtain a second data traffic set, wherein the second data traffic set includes second data traffic generated by multiple services accessing the second access address within a second time period, each second data traffic carries the service identifier, and the second time period is the time period for data synchronization between the source database and the target database; If the second data traffic in the second data traffic set already carries the multiple service identifiers, then it is determined that the first access address has been completely switched to the second access address; If the data synchronization proceeds normally, a first parameter is set on the source database, wherein the first parameter instructs the source database to stop writing operations and instructs the source database to disconnect the data connection with the service indicated by the service identifier in the service identifier set; After the source database and the target database have completed data synchronization, a second parameter is set on the source database, wherein the second parameter instructs the source database to stop read / write operations; If the data connection between the source database and the service indicated by the service identifier in the service identifier set is a preset value, then the second access address is changed to point to the target database.

2. The method according to claim 1, characterized in that, The data synchronization between the source database and the target database includes: Backup data is obtained from the source database and imported into the target database. The incremental data from the source database is synchronized to the target database in real time to complete the data synchronization between the source database and the target database.

3. The method according to claim 1, characterized in that, The acquisition of the first data traffic set includes: A data traffic acquisition instruction is issued, wherein the data traffic acquisition instruction instructs the source database to acquire the data traffic accessing the first access address within a first time period; The data traffic obtained from the data traffic acquisition instruction is used as the first data traffic set.

4. The method according to claim 3, characterized in that, Each data traffic in the data traffic set includes the service source address; The step of determining the service identifier set based on the first data traffic set includes: Based on the access source address included in each data traffic in the data traffic set, determine the access curve for accessing the first access address within the first time period; Based on the access curve of accessing the first access address within the first time period, the multiple services that access the first access address within the first time period are determined to obtain the service identifier set.

5. A data processing apparatus, characterized in that, The data processing device includes: The acquisition module is used to acquire a first data traffic set, wherein the first data traffic set includes first data traffic generated by multiple services accessing a first access address within a first time period, the first access address belongs to a source database and points to the source database, and each first data traffic carries a service identifier, the service identifier indicating the service. The determining module is used to determine a service identifier set based on the first data traffic set, wherein the service identifier set includes multiple service identifiers, the service identifiers indicate services that access the first access address within the first time period, and each service identifier corresponds to the first access address; The update module is used to change the first access address corresponding to each service identifier in the service identifier set to a second access address when the source database and the target database are synchronizing data, wherein the second access address belongs to the target database and the second access address points to the source database; The configuration module is used to obtain a second data traffic set, wherein the second data traffic set includes second data traffic generated by multiple services accessing the second access address within a second time period, each second data traffic carrying the service identifier, and the second time period being the time period for data synchronization between the source database and the target database; if the second data traffic in the second data traffic set already carries the multiple service identifiers, then it is determined that the first access address has been completely switched to the second access address; if the data synchronization is proceeding normally, then a first parameter is set on the source database, wherein the first parameter instructs the source database to stop write operations and instructs the source database to disconnect the data connection with the service indicated by the service identifier in the service identifier set; The update module is further configured to set a second parameter on the source database after the source database and the target database have completed data synchronization, wherein the second parameter indicates that the source database stops read / write operations; if the data connection between the source database and the service indicated by the service identifier in the service identifier set is a preset value, then the second access address is changed to point to the target database.

6. The apparatus according to claim 5, characterized in that, The device further includes a synchronization module, which is used for: Backup data is obtained from the source database and imported into the target database. The incremental data from the source database is synchronized to the target database in real time to complete the data synchronization between the source database and the target database.

7. The apparatus according to claim 5, characterized in that, The acquisition module is specifically used for: A data traffic acquisition instruction is issued, wherein the data traffic acquisition instruction instructs the source database to acquire the data traffic accessing the first access address within a first time period; The data traffic obtained from the data traffic acquisition instruction is used as the first data traffic set.

8. The apparatus according to claim 7, characterized in that, Each data traffic in the data traffic set includes the service source address; The determining module is specifically used for: Based on the access source address included in each data traffic in the data traffic set, determine the access curve for accessing the first access address within the first time period; Based on the access curve of accessing the first access address within the first time period, the multiple services that access the first access address within the first time period are determined to obtain the service identifier set.

9. A computer device, characterized in that, include: Memory, transceiver, processor, and bus system; The memory is used to store programs; The processor is used to execute the program in the memory to implement the method of any one of claims 1 to 4; The bus system is used to connect the memory and the processor to enable communication between the memory and the processor.

10. A computer-readable storage medium, characterized in that, Includes instructions that, when run on a computer, cause the computer to perform the method as described in any one of claims 1 to 4.

11. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the method as described in any one of claims 1 to 4.

Images