Data processing method, device, computer equipment and storage medium

Abstract

The present application discloses a data processing method, device, computer equipment and storage medium, including: obtaining source data to be processed; setting the server storing the source data as the master server of a preset distributed system, setting the remaining servers of the distributed system as slave servers of the master server, and the master server synchronizing the source data to the slave servers by copying; the master server and / or the slave server perform hash calculations on the user data one by one according to a preset hash function, and store the corresponding user data in the storage location pointed to by the hash calculation result; configuring the query interface of the source data. When storing the source data, the server that collects the source data is set as the master server in the distributed system, and the master server copies the synchronized source data to the remaining slave servers, so that the distributed system can improve the data processing efficiency of the entire source data call.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data processing, and in particular to a data processing method, apparatus, computer equipment and storage medium. Background Art

[0002] Data storage refers to temporary files generated during the processing of data flow or information that needs to be found during the processing. Data is recorded in a certain format inside the computer or on an external storage medium. Data storage must be named, and this naming must reflect the composition and meaning of information characteristics. Data flow reflects the data flowing in the system and shows the characteristics of dynamic data; data storage reflects the static data in the system and shows the characteristics of static data.

[0003] The inventor of the present invention found in the research that in the prior art, when storing data, the data to be stored is often stored in one service. When storing, the information is first stored in a specified location, and a storage list is established to record the storage location of each information. This single storage method will cause the server storing data to have a large access pressure, and when calling data, it is necessary to traverse the storage location and then call the data. When the stored data is large, the calling efficiency is poor. Summary of the invention

[0004] The embodiments of the present invention provide a data processing method, device, computer equipment and storage medium capable of improving data storage efficiency and call efficiency.

[0005] In order to solve the above technical problems, a technical solution adopted by an embodiment of the present invention is to provide a data processing method, comprising:

[0006] Acquire source data to be processed, wherein the source data consists of multiple pieces of user data;

[0007] The server storing the source data is set as a master server of a preset distributed system, and the remaining servers of the distributed system are set as slave servers of the master server, and the master server synchronizes the source data to the slave servers by replication;

[0008] The master server and / or the slave server performs hash calculations on the user data one by one according to a preset hash function, and stores the corresponding user data in a storage location pointed to by the hash calculation result;

[0009] The query interface of the source data is configured so that an external terminal can call the source data.

[0010] Optionally, the master server synchronizing the source data to the slave server by replication includes:

[0011] The main server extracts the source data into memory;

[0012] The source data is cleaned according to preset data cleaning rules, and the master server synchronizes the cleaned source data to each slave server of the distributed system.

[0013] Optionally, the master server and / or the slave server performs hash calculation on the user data one by one according to a preset hash function, and stores the corresponding user data in a storage location pointed to by the hash calculation result, including:

[0014] The master server and / or the slave server calls a preset hash function;

[0015] Performing hash calculation on the user data one by one to generate a hash value, and performing duplicate checking on the user data according to the hash value;

[0016] The user data with the same hash value is deleted to make each user data in the source data unique.

[0017] Optionally, deleting the user data with the same hash value includes:

[0018] Compare the user data with the same hash value to determine target user data with the same hash value but different user data;

[0019] Constructing a storage linked list for storing the target user data according to the target user data;

[0020] The storage linked list is associated and stored with the storage location pointed to by the hash value in a key value manner;

[0021] The user data having the same hash value and the same user data is deleted.

[0022] Optionally, the distributed system includes a plurality of sentinel nodes, and the plurality of sentinel nodes are connected to the master server and a plurality of slave servers. After configuring the query interface of the source data, the configuration includes:

[0023] The multiple sentinel nodes send network packet discovery commands to the master server and the multiple slave servers respectively;

[0024] According to the reply information of the main server to the network packet exploration command, confirm whether the main server is offline subjectively;

[0025] When the master server is in a subjective offline state, the multiple sentinel nodes perform joint verification to determine whether the master server is objectively offline. When the master server is in an objective offline state, a new master server is selected from the multiple slave servers.

[0026] Optionally, after configuring the query interface of the source data, the following steps are included:

[0027] The master server and / or the slave server divides the source data into memory data and hard disk data according to a preset configuration ratio;

[0028] The master server and / or slave server pre-load the memory data into their respective memory spaces.

[0029] Optionally, after configuring the query interface of the source data, the following steps are included:

[0030] Obtaining a user request to be executed, wherein the user request carries user information, and the user information is used to retrieve at least one piece of user data corresponding thereto;

[0031] Performing hash calculation on the user information according to the hash function to generate a query address;

[0032] calling at least one piece of user data corresponding to the user information according to the query address;

[0033] The at least one piece of user data is transmitted back to the user terminal that sent the user request.

[0034] In order to solve the above technical problems, an embodiment of the present invention further provides a data processing device, including:

[0035] An acquisition module, used to acquire source data to be processed, wherein the source data consists of multiple pieces of user data;

[0036] A processing module, configured to set the server storing the source data as a master server of a preset distributed system, and set the remaining servers of the distributed system as slave servers of the master server, wherein the master server synchronizes the source data to the slave servers by replication;

[0037] A storage module, used for the master server and / or the slave server to perform hash calculations on the user data one by one according to a preset hash function, and store the corresponding user data in a storage location pointed to by the hash calculation result;

[0038] The configuration module is used to configure the query interface of the source data so that an external terminal can call the source data.

[0039] Optionally, the data processing device further includes:

[0040] A first extraction submodule, used for the main server to extract the source data into a memory;

[0041] The first cleaning submodule is used to clean the source data according to a preset data cleaning rule, and the master server synchronizes the cleaned source data to each slave server of the distributed system.

[0042] Optionally, the data processing device further includes:

[0043] A first calling submodule, used for the master server and / or the slave server to call a preset hash function;

[0044] A first calculation submodule, configured to perform hash calculation on the user data one by one to generate a hash value, and perform duplicate checking on the user data according to the hash value;

[0045] The first deleting submodule is used to delete the user data with the same hash value, so that each user data in the source data is unique.

[0046] Optionally, the data processing device further includes:

[0047] A first comparison submodule, used to compare the user data with the same hash value to determine target user data with the same hash value but different user data;

[0048] A first storage submodule, configured to construct a storage linked list for storing the target user data according to the target user data;

[0049] A first associating submodule, used for associating and storing the storage linked list with the storage location pointed to by the hash value in a key value manner;

[0050] The second deleting submodule is used to delete the user data having the same hash value and the same user data.

[0051] Optionally, the distributed system includes a plurality of sentinel nodes, the plurality of sentinel nodes are connected to the master server and the plurality of slave servers, and the data processing device further includes:

[0052] A first sending submodule, configured for the plurality of sentinel nodes to send network packet discovery commands to the master server and the plurality of slave servers respectively;

[0053] A first confirmation submodule, used to confirm whether the main server is subjectively offline according to the reply information of the main server to the network packet exploration command;

[0054] The first execution submodule is used to, when the master server is in a subjective offline state, perform joint verification through the multiple sentinel nodes to determine whether the master server is objectively offline, and when the master server is in an objective offline state, select a new master server from the multiple slave servers.

[0055] Optionally, the data processing device further includes:

[0056] A first division submodule, configured for the master server and / or the slave server to divide the source data into memory data and hard disk data according to a preset configuration ratio;

[0057] The second extraction submodule is used for the master server and / or the slave server to pre-extract the memory data into their respective memory spaces.

[0058] Optionally, the data processing device further includes:

[0059] A first acquisition submodule is used to acquire a user request to be executed, wherein the user request carries user information, and the user information is used to retrieve at least one piece of user data corresponding thereto;

[0060] A first processing submodule, configured to perform a hash calculation on the user information according to the hash function to generate a query address;

[0061] A second calling submodule, configured to call at least one piece of user data corresponding to the user information according to the query address;

[0062] The second execution submodule is used to transmit the at least one piece of user data back to the user terminal that sends the user request.

[0063] To solve the above technical problem, an embodiment of the present invention further provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the above data processing method.

[0064] In order to solve the above technical problem, an embodiment of the present invention further provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the above data processing method.

[0065] The beneficial effects of the embodiments of the present invention are: when storing source data, the server that collects the source data is set as the master server in the distributed system, and the master server is used to copy and synchronize the source data to the remaining slave servers. The distributed system can improve the data processing efficiency of the entire source data call and the security of the source data storage. By setting the master server to synchronize data, the data synchronization efficiency can be higher, and the problem of repeated data synchronization is avoided when data crosses or is broadcast and diffused. The hash function is used to perform hash calculation on the user data in the source data, and the user data is stored by the result of the hash calculation, so that when the source data is stored, there is no need to establish a storage list. At the same time, when calling the source data, there is no need to traverse the list, which greatly improves the efficiency of source data storage and query calling. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0067] Figure 1 A basic flow chart of a data processing method according to a specific embodiment of the present application;

[0068] Figure 2 A schematic diagram of a process for cleaning and synchronizing source data according to a specific embodiment of the present application;

[0069] Figure 3 A schematic diagram of a process for determining the uniqueness of user data according to a specific embodiment of the present application;

[0070] Figure 4 A schematic diagram of a process for deleting user data according to a specific embodiment of the present application;

[0071] Figure 5 A schematic diagram of a process of replacing a main server according to a specific embodiment of the present application;

[0072] Figure 6 A schematic diagram of a process of preheating and storing source data according to a specific embodiment of the present application;

[0073] Figure 7 A flowchart of replying to a user request according to a specific embodiment of the present application is shown;

[0074] Figure 8 A schematic diagram of the basic structure of a data processing device according to an embodiment of the present application;

[0075] Fig. 9 This is a basic structural block diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0076] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0077] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0078] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0079] It will be understood by those skilled in the art that the term "terminal" as used herein includes both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of performing two-way communications over a two-way communication link. Such devices may include: cellular or other communication devices, which have a single-line display or a multi-line display or a cellular or other communication device without a multi-line display; PCS (Personal Communications Service), which may combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant), which may include a radio frequency receiver, a pager, Internet / intranet access, a web browser, a notepad, a calendar and / or a GPS (Global Positioning System) receiver; conventional laptop and / or palmtop computers or other devices, which have and / or include a conventional laptop and / or palmtop computer or other device with and / or including a radio frequency receiver. The "terminal" used here can be portable, transportable, installed in a vehicle (air, sea and / or land), or suitable for and / or configured to run locally, and / or in a distributed form, running at any other location on the earth and / or in space. The "terminal" used here can also be a communication terminal, an Internet terminal, a music / video playing terminal, for example, a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or a smart TV, a set-top box and other devices.

[0080] See also Figure 1 , Figure 1 Schematic diagram of the basic flow of the data processing method of this embodiment.

[0081] like Figure 1 As shown, a data processing method includes:

[0082] S1100, obtaining source data to be processed, wherein the source data is composed of multiple pieces of user data;

[0083] In this implementation, source data refers to unprocessed data that needs to be stored. The source of source data can be: log files generated by system operation, data files uploaded by terminals, or user data in the system.

[0084] In some implementations, when the data processing method of this embodiment is applied to user risk management of a webpage system, a live broadcast platform, or a short video platform, the types of source data include anchor tag data, value data, image data, AI recognition anchor feature data, operational custom data in MYSQL, anchor UID or channel SID, etc., and user data is user data formed by storing various types of data such as an associated image data, feature string data, and tag data into a unified data structure. It should be pointed out that both source data and user data are data definitions of a type of data. Depending on the specific application scenario, user data can be extended to any data type, and source data can be extended to a data set of any data type.

[0085] S1200: setting a server storing the source data as a master server of a preset distributed system, and setting other servers of the distributed system as slave servers of the master server, wherein the master server synchronizes the source data to the slave servers by replication;

[0086] In this embodiment, data is processed through a distributed system, which includes several interconnected server components, and the collection of source data occurs in a server in the distributed system. After a server collects the source data, it sends a sharing request to the system. After receiving the sharing request, the system sets the server as the master server. The master server has the authority to share data with other servers. After the master server is set, other servers in the distributed system are configured as slave servers of the master server, and the slave servers can synchronize data from the master server.

[0087] When a master server is selected in a distributed system and other servers are configured as slave servers of the master server, the master server begins to synchronize data with the slave servers by replication. The result of the synchronization is that the source data stored in the master server is copied to each slave server. After the slave server obtains the source data through synchronization, it can accept call requests for the source data or user data therein in the area where it is located, which greatly reduces the pressure on data calls on the master server.

[0088] The master-slave server data synchronization method can effectively reduce the trouble of repeated data verification caused by the same server node receiving data requests from multiple server nodes when data synchronization is performed through broadcast diffusion in traditional distributed systems, further improving the efficiency of data synchronization.

[0089] S1300: The master server and / or the slave server performs hash calculations on the user data one by one according to a preset hash function, and stores the corresponding user data in a storage location pointed to by the hash calculation result;

[0090] After the master server and the slave server complete data synchronization, they need to store the source data separately. The storage method is as follows: each server node performs hash operation on the user data in the source data one by one through a uniformly set hash formula. The hash operation obtains the hash value of each user data. The hash value maps to a physical storage location, and the corresponding user data is stored in the physical storage location mapped by the hash value. When there are multiple types of data in the user data, the hash operation can be performed on one data type or all data types separately. The data stored in this way can accurately locate the storage location of the user data under limited retrieval conditions when performing data query.

[0091] In some embodiments, when performing data storage, not all server nodes need to use hash function operations for storage. In some embodiments, in the master server or slave server, historical data confirms that the number of calls to source data by the server node is very small and does not reach the set threshold, or the server is not responsible for calling the source data in the distributed system. These master servers or slave servers do not need to perform hash operations on user data and can directly use traditional data storage methods for storage.

[0092] S1400: Configure a query interface for the source data so that an external terminal can call the source data.

[0093] After storage is completed, a query interface is configured for the source data. The query interface can be an HTTP or RPC interface. The RPC interface is a protocol that requests services from a remote computer program over the network without having to understand the underlying network technology, and can achieve more efficient and secure data transmission.

[0094] In some embodiments, in a distributed system, the calling interfaces of the master server and the slave server for the source file are configured as different interfaces, and the configuration method is to obtain the interface form of each server's high-frequency call based on historical data statistics, and determine the calling interface type of the source server based on the high-frequency interface form used by the server. However, the setting of the interface type is not limited to this. According to different specific application scenarios, in some embodiments, the interface configuration of the master server and the slave server can be configured as a unified interface.

[0095] When the interface configuration for the source data is completed on each server, the source data can be called by an external terminal, which can be a mobile terminal or a PC terminal, or a business server terminal that executes a specific transaction.

[0096] In the above implementation, when storing source data, the server that collects the source data is set as the master server in the distributed system, and the master server copies and synchronizes the source data to the remaining slave servers. The distributed system can improve the data processing efficiency of the entire source data call and the security of the source data storage. By setting the master server to synchronize data, the data synchronization efficiency can be higher, and the problem of repeated data synchronization is avoided when data crosses or is broadcast and diffused. The hash function is used to perform hash calculation on the user data in the source data, and the user data is stored by the result of the hash calculation, so that when the source data is stored, there is no need to establish a storage list. At the same time, when calling the source data, there is no need to traverse the list, which greatly improves the efficiency of source data storage and query calls.

[0097] In some implementations, before storing the source data, it is necessary to clean the source data and synchronize the cleaned source data. Figure 2 , Figure 2 The figure is a flow chart of data cleaning and synchronization of source data in this embodiment.

[0098] like Figure 2 As shown, S1200 includes:

[0099] S1211, the main server extracts the source data into memory;

[0100] After the main server is determined in the distributed system, the main server stores the source data in the memory space. In some embodiments, after the source data is collected, it is stored in the hard disk of the main server. When it is determined that the source data is to be shared, the main server calls the source data from the hard disk space to the memory space. In other embodiments, when the source data is stored in the main server, part of the data is stored in the memory space and part of the data is stored in the hard disk space. When it is determined that the source data is to be shared, the main server calls the remaining source data from the hard disk space to the memory space.

[0101] S1212: Clean the source data according to preset data cleaning rules, and the master server synchronizes the cleaned source data to each slave server of the distributed system.

[0102] The source data in the memory space is cleaned by the set cleaning rules, and the data cleaning method is to filter the source data by whitelist or blacklist. Specifically, when the data cleaning rule is a whitelist, the user data in the source data that is in the whitelist is retained; when the data cleaning rule is a blacklist, the user data in the source data that is in the blacklist is deleted. However, the data cleaning rules are not limited to this. According to different specific application scenarios, in some embodiments, the cleaning rules can be used to filter the user data containing sensitive information in the source data by sensitive information.

[0103] After the master server cleans the source data, it starts synchronizing the cleaned source data to each slave server. By cleaning and synchronizing the source data with the master server, the slave servers can avoid repeated data cleaning, thus improving the efficiency of data cleaning.

[0104] In some implementations, when the server stores source data, it is necessary to delete duplicate data in order to improve the efficiency of data storage and query. Figure 3 , Figure 3 The figure is a schematic diagram of a process for determining the uniqueness of user data in this embodiment.

[0105] like Figure 3 As shown, S1300 includes:

[0106] S1311, the master server and / or the slave server calls a preset hash function;

[0107] After completing data synchronization between the master server and the slave server, each needs to store the source data. The storage method is: each server node uniformly sets a hash formula. When storing, each server calls the hash formula stored locally to calculate the user data.

[0108] S1312, performing hash calculation on the user data one by one to generate a hash value, and performing duplicate checking on the user data according to the hash value;

[0109] Hash operations are performed on the user data in the source data one by one to obtain the hash value of each user data. After the hash value is obtained, the hash value is compared with the previously generated hash value. When the comparison confirms that the same hash value already exists, the user data corresponding to the hash value is confirmed to be duplicate data.

[0110] S1313. Delete the user data with the same hash value to make each user data in the source data unique.

[0111] Delete user data with the same hash value to make the user data in the source data unique. Storing user data with the same hash value can effectively ensure that each hash value is unique when the source data is hashed and stored, and each hash value can uniquely locate a storage address. Map all user data into HASHMAP and output the HASHMAP to REDIS, which can support 100,000 read and write performances per second and provide O1 query capabilities.

[0112] When storing data through hash values, when the amount of data is too large, different user data may generate the same hash value after hash operation. At this time, a problem is that when more than two user data are mapped to the same hash value, and the hash value is mapped to the same storage address, there will be a problem of two different user data stored in the same storage location. By deleting user data with the same hash value through hash values, the problem of duplicate hash values ​​is avoided, the accuracy of storage is improved, and the query efficiency is improved at the same time.

[0113] In some implementations, there are two scenarios when the hash values ​​of user data are the same: first, the two user data are exactly the same; second, the two user data are different, but the final hash values ​​are the same. For the above two different scenarios, different processing methods are used for user data with the same hash value. Figure 4 , Figure 4 This is a schematic diagram of the process of deleting user data in this embodiment.

[0114] like Figure 4 As shown, S1313 includes:

[0115] S1321, performing data comparison on the user data with the same hash value to determine target user data with the same hash value but different user data;

[0116] When it is determined that there is user data with the same hash value in the source data, the user data with the same hash value is compared. The comparison method can be to calculate the Hamming distance or Hamming distance of the two user data.

[0117] By comparing user data with the same hash value, target user data with the same hash value but different user data is determined.

[0118] S1322, constructing a storage linked list for storing the target user data according to the target user data;

[0119] After determining the target user data with the same hash value but different user data, it can be determined that two different user data with the same hash value will be stored in the same storage location. At this time, the problem of storage collision arises. In order to solve this problem, we need to create a storage linked list and store one of the user data in the storage linked list.

[0120] S1323, associating the storage linked list with the storage location pointed to by the hash value by means of a key value;

[0121] After the storage linked list is established, one of the user data is stored in the storage linked list, and the linked list and the storage location of the hash value mapping are associated with a key-value pair, that is, a key is established between the storage location and the linked list. When querying, after determining the storage location through the hash value, it is necessary to check whether there is a storage linked list at the location. When storing the storage linked list, it is necessary to match the search conditions with the storage content of the linked list. When the match is successful, the content in the linked list is read as the query result.

[0122] When there are multiple user data with the same hash value but different data contents, a storage linked list with the same number of user data and connected in sequence is created. Each storage linked list stores one user data. Key-value pairs are built between adjacent linked lists for associated storage. A key-value pair is built between the first linked list and the storage location for associated storage. When querying, the content of a series of storage linked lists pointed to by the storage location is traversed to find the user data that meets the conditions.

[0123] S1324. Delete the user data having the same hash value and the same user data.

[0124] By comparing and determining the user data with the same hash value and the same user data in the data source, the above user data is deleted, leaving only the first user data with the hash value, to ensure the uniqueness of the user data in the system. In some implementations, it is also possible to select one user data to retain by random selection and delete other identical user data.

[0125] In some embodiments, a sentinel node is provided in the distributed system. The sentinel node user checks the running status of the master server and each slave server in the distributed system. Each sentinel node can monitor multiple master servers. Therefore, there is a situation where one master server is multiplexed and checked by multiple sentinel nodes. Figure 5 , Figure 5 The figure is a flow chart of replacing the main server in this embodiment.

[0126] like Figure 5 As shown, S1400 and later include:

[0127] S1411, the multiple sentinel nodes send network packet exploration commands to the master server and the multiple slave servers respectively;

[0128] Each sentinel node in the distributed system sends a ping command (network packet discovery command) to the master server and slave server connected to it. This command is a network connection command used to check whether each master server and slave server are in working state. The sentinel node sends a ping command to each master server and slave server at a timed interval, for example, once a second. After receiving the command, each server needs to send a reply to the sentinel.

[0129] S1412, confirming whether the main server is offline subjectively according to the reply information of the main server to the network packet discovery command;

[0130] Each sentinel node determines whether the main server is online based on the reply information of the main server to the ping command. If the main server can send a reply message to the sentinel node, the sentinel node determines that the main server is online. If the main server does not send a reply message back to the sentinel node, the sentinel node determines that the main server is offline.

[0131] However, at this time, the offline status of the main server determined cannot rule out whether it is caused by network transmission obstacles or the main server is really offline. Therefore, the offline status determined by a single sentinel node is defined as subjective offline.

[0132] S1413. When the master server is in a subjective offline state, a joint verification is performed through the multiple sentinel nodes to determine whether the master server is objectively offline. When the master server is in an objective offline state, a new master server is selected from the multiple slave servers.

[0133] When it is determined that the main server is in a subjective offline state, the sentinel node needs to collect other sentinel nodes connected to the main server. At this time, multiple sentinel nodes connected to the subjective server will perform a joint verification. The method of joint verification is: multiple sentinel nodes vote on whether the main server is in an offline state. Each sentinel node votes according to whether the main server is in a subjective offline state. When the voting result shows that the number of sentinel nodes that determine that the subjective server is in a subjective offline state reaches the set number, it is determined that the main server is in an objective offline state, that is, the system determines that the subjective server is already offline.

[0134] When it is determined that the master server is offline, a server needs to be selected from the slave servers to become the new master server. The selection method can be: random selection, or it can be selected by verifying whether the slave servers have complete source data. When multiple slave servers have complete source data, a new master server is selected by random selection. In some embodiments, when selecting a master server, the slave server with the best computing performance or the lowest occupancy rate is selected from the slave servers as the master server.

[0135] In some implementations, in order to ensure that each server in the distributed system can respond to user needs as quickly as possible while not occupying too much memory resources when working, the storage status of the source data needs to be adjusted after the source data is stored. Figure 6 , Figure 6 The figure is a flow chart of the source data preheating storage in this embodiment.

[0136] like Figure 6 As shown, S1400 and later include:

[0137] S1421, the master server and / or the slave server divides the source data into memory data and hard disk data according to a preset configuration ratio;

[0138] After the source data is stored, the master server and each slave server need to store the source data separately according to their own business needs. The differentiated storage method is: store some hot data in the memory space, and store some infrequently used or relatively infrequently used data in the hard disk space.

[0139] In this embodiment, the source data can be divided in a way of agreeing to divide. According to the historical information statistics, the data that is most frequently called or requested in the historical data is divided into memory data stored in the memory space, and the remaining data is divided into hard disk data. In this embodiment, the main server and each slave server divide the source data in the same way. However, the data division method is not limited to this. According to different specific application scenarios, in some embodiments, each server in the distributed system counts the frequently used data in the historical data according to its own business situation, and then divides the source data according to their own statistical situation, and then stores their own memory data and hard disk data respectively according to the division results.

[0140] S1422. The master server and / or slave server pre-loads the memory data into their respective memory spaces.

[0141] After the master server and each slave server in the distributed system divide the source data into memory data and hard disk data, the master server and each slave server extract the memory data into the memory space for storage.

[0142] In some implementations, since the main server calls the source data at a high frequency, only the main server divides and stores the source data.

[0143] In some implementations, the main server does not connect to the external terminal's call for source data. When the slave server connects to the specific external call requirements, the slave server only divides and stores the source data.

[0144] Preheating and dividing the storage of source data can improve the system's response speed to external call requests.

[0145] In some implementations, when a user request is received, the user data requested by the user needs to be called by querying. Figure 7 , Figure 7 The figure is a flowchart of replying to a user request in this embodiment.

[0146] like Figure 7 As shown, S1400 and later include:

[0147] S1431. Obtain a user request to be executed, wherein the user request carries user information, and the user information is used to retrieve at least one piece of user data corresponding thereto;

[0148] After the source data is stored in the master server and the slave server, it is necessary to receive calls from external demands. The calling method is: get the user request of the user. The sender of the user request can be a mobile terminal, a PC terminal or a business server terminal. The user request carries the user information of the requested data, where the user information is equivalent to one or more user data in the source data, or local information in the user data. For example, when the user data includes: the anchor's label data, value data, picture data, AI recognition anchor feature data, operational custom data in MYSQL, anchor UID or channel SID and other data categories, the request information only contains the anchor UID information.

[0149] S1432, performing hash calculation on the user information according to the hash function to generate a query address;

[0150] When a user request is received and the user information is extracted, a hash operation is performed on the user information through the hash function of the main server or the slave server to obtain a hash value corresponding to the user information. Since the user information includes local data information in the user data, the hash value calculated based on the user information is included in the hash value during storage. The storage location of at least one corresponding user data can be determined through the hash value of the user information, and the corresponding user data can be read from the storage location.

[0151] In some implementations, when the user information includes multiple user data retrieval requirements, the retrieval requirements need to be disassembled, and then the corresponding hash values ​​and storage locations are determined one by one.

[0152] S1433. Call at least one piece of user data corresponding to the user information according to the query address;

[0153] According to the query address determined by the hash value, the user data stored in the query address is called. When there are multiple query addresses, multiple user data corresponding to different query addresses are read respectively.

[0154] In some implementations, when a storage linked list with key-value pairs exists at the query address, it is necessary to traverse and compare the user information with the data in the storage linked list to determine the user data stored in the linked list with the highest matching degree, which is the user data corresponding to the user information.

[0155] S1434. Transmit the at least one piece of user data back to the user terminal that sent the user request.

[0156] After the user data corresponding to the user information is determined, the user data is sent back to the user terminal that sent the user request, and the call to the user data is completed. Responding to the user's query request by HASHMAP can support 100,000 read and write performances per second and provide O1 query capability.

[0157] The present application can construct a corresponding device by running the application program of each embodiment of the aforementioned method in a computer. For details, please refer to Figure 8 , Figure 8 Schematic diagram of the basic structure of the data processing device of this embodiment.

[0158] like Figure 8As shown, a data processing device includes: an acquisition module 2100, a processing module 2200, a storage module 2300 and a configuration module 2400. The acquisition module 2100 is used to acquire source data to be processed, wherein the source data is composed of multiple user data; the processing module 2200 is used to set the server storing the source data as the master server of a preset distributed system, and set the remaining servers of the distributed system as slave servers of the master server, and the master server synchronizes the source data to the slave servers by replication; the storage module 2300 is used for the master server and / or the slave server to perform hash calculations on the user data one by one according to a preset hash function, and store the corresponding user data in the storage location pointed to by the hash calculation result; the configuration module 2400 is used to configure the query interface of the source data so that an external terminal can call the source data.

[0159] When the data processing device is storing source data, the server that collects the source data is set as the master server in the distributed system, and the master server is used to copy and synchronize the source data to the remaining slave servers. The distributed system can improve the data processing efficiency of the entire source data call and the security of the source data storage. By setting the master server to synchronize data, the data synchronization efficiency can be higher, and the problem of repeated data synchronization is avoided when the data is crossed or broadcast. The hash function is used to perform hash calculation on the user data in the source data, and the user data is stored by the result of the hash calculation, so that when the source data is stored, there is no need to establish a storage list. At the same time, when the source data is called, there is no need to traverse the list, which greatly improves the efficiency of source data storage and query calls.

[0160] In some embodiments, the data processing device further includes: a first extraction submodule and a first cleaning submodule. The first extraction submodule is used by the master server to extract the source data into the memory; the first cleaning submodule is used to clean the source data according to the preset data cleaning rules, and the master server synchronizes the cleaned source data to each slave server of the distributed system.

[0161] In some embodiments, the data processing device further includes: a first calling submodule, a first calculating submodule and a first deleting submodule. The first calling submodule is used for the master server and / or the slave server to call a preset hash function; the first calculating submodule is used to perform hash calculations on the user data one by one to generate hash values, and perform duplicate checking on the user data according to the hash values; the first deleting submodule is used to delete the user data with the same hash value, so that each user data in the source data is unique.

[0162] In some embodiments, the data processing device further includes: a first comparison submodule, a first storage submodule, a first association submodule, a second deletion submodule. The first comparison submodule is used to compare the user data with the same hash value to determine the target user data with the same hash value but different user data; the first storage submodule is used to construct a storage linked list for storing the target user data according to the target user data; the first association submodule is used to associate the storage linked list with the storage location pointed to by the hash value by means of a key value; the second deletion submodule is used to delete the user data with the same hash value and the same user data.

[0163] In some embodiments, the distributed system includes a plurality of sentinel nodes, and the plurality of sentinel nodes are connected to the master server and a plurality of slave servers, and the data processing device further includes: a first sending submodule, a first confirmation submodule, and a first execution submodule. Among them, the first sending submodule is used for the plurality of sentinel nodes to send network packet exploration commands to the master server and the plurality of slave servers respectively; the first confirmation submodule is used to confirm whether the master server is subjectively offline according to the reply information of the master server to the network packet exploration command; the first execution submodule is used for, when the master server is in a subjective offline state, to perform joint verification through the plurality of sentinel nodes to determine whether the master server is objectively offline, and when the master server is in an objective offline state, to select a new master server from the plurality of slave servers.

[0164] In some embodiments, the data processing device further includes: a first division submodule and a second extraction submodule. The first division submodule is used by the master server and / or the slave server to divide the source data into memory data and hard disk data according to a preset configuration ratio; the second extraction submodule is used by the master server and / or the slave server to pre-draw the memory data into their respective memory spaces.

[0165] In some embodiments, the data processing device further includes: a first acquisition submodule, a first processing submodule, a second calling submodule, and a second execution submodule. The first acquisition submodule is used to acquire a user request to be executed, wherein the user request carries user information, and the user information is used to retrieve at least one piece of user data corresponding thereto; the first processing submodule is used to generate a query address by performing a hash calculation on the user information according to the hash function; the second calling submodule is used to call at least one piece of user data corresponding to the user information according to the query address; and the second execution submodule is used to transmit the at least one piece of user data back to the user terminal that sent the user request.

[0166] In order to solve the above technical problems, the embodiment of the present application also provides a computer device for running a computer program implemented according to the data processing method. Figure 8 , Fig. 9 This is a basic structural block diagram of the computer device in this embodiment.

[0167] like Fig. 9 As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a non-volatile storage medium, a memory, and a network interface connected via a system bus. Among them, the non-volatile storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor may implement a data processing method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor may execute a data processing method. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0168] In this embodiment, the processor is used to execute Figure 8 The specific functions of the acquisition module 2100, the processing module 2200, the storage module 2300 and the configuration module 2400 are stored in the memory, and the program code and various data required to execute the above modules are stored. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all sub-modules in the data processing device, and the server can call the program code and data of the server to execute the functions of all sub-modules.

[0169] When the computer device is storing source data, the server that collects the source data is set as the master server in the distributed system, and the master server is used to copy and synchronize the source data to the remaining slave servers. The distributed system can improve the data processing efficiency of the entire source data call and the security of the source data storage. By setting the master server to synchronize data, the data synchronization efficiency can be higher, and the problem of repeated data synchronization is avoided when the data is crossed or broadcast. The hash function is used to perform hash calculation on the user data in the source data, and the user data is stored by the result of the hash calculation, so that the source data does not need to establish a storage list when storing data, and at the same time, when calling the source data, there is no need to traverse the list, which greatly improves the efficiency of source data storage and query calls.

[0170] The present application also provides a non-volatile storage medium, in which the data processing method is written into a computer program and stored in the storage medium in the form of computer-readable instructions. When the computer-readable instructions are executed by one or more processors, it means that the program is run in the computer, thereby enabling one or more processors to execute the steps of the data processing method of any of the above-mentioned embodiments.

[0171] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0172] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0173] Those skilled in the art will appreciate that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, altered, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted.

[0174] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: include: Acquire source data to be processed, wherein the source data consists of multiple pieces of user data; The server storing the source data is set as a master server of a preset distributed system, and the remaining servers of the distributed system are set as slave servers of the master server, and the master server synchronizes the source data to the slave servers by replication, wherein the distributed system includes a plurality of sentinel nodes, and the plurality of sentinel nodes are connected to the master server and the plurality of slave servers; The master server and / or the slave server performs hash calculations on the user data one by one according to a preset hash function, and stores the corresponding user data in a storage location pointed to by the hash calculation result; Configuring a query interface for the source data so that an external terminal can call the source data; The multiple sentinel nodes send network packet discovery commands to the master server and the multiple slave servers respectively; According to the reply information of the main server to the network packet exploration command, confirm whether the main server is offline subjectively; When the master server is in a subjective offline state, the multiple sentinel nodes perform joint verification to determine whether the master server is objectively offline. When the master server is in an objective offline state, a new master server is selected from the multiple slave servers.

2. The data processing method according to claim 1, characterized in that: The master server synchronizes the source data to the slave server by replication, including: The main server extracts the source data into memory; The source data is cleaned according to preset data cleaning rules, and the master server synchronizes the cleaned source data to each slave server of the distributed system.

3. The data processing method according to claim 1, characterized in that: The master server and / or the slave server performs hash calculations on the user data one by one according to a preset hash function, and stores the corresponding user data in a storage location pointed to by the hash calculation result, including: The master server and / or the slave server calls a preset hash function; Performing hash calculation on the user data one by one to generate a hash value, and performing duplicate checking on the user data according to the hash value; The user data with the same hash value is deleted to make each user data in the source data unique.

4. The data processing method according to claim 3, characterized in that: Deleting the user data having the same hash value includes: Compare the user data with the same hash value to determine target user data with the same hash value but different user data; Constructing a storage linked list for storing the target user data according to the target user data; The storage linked list is associated and stored with the storage location pointed to by the hash value in a key value manner; The user data having the same hash value and the same user data is deleted.

5. The data processing method according to claim 1, characterized in that: After configuring the query interface of the source data, the following steps are included: The master server and / or the slave server divides the source data into memory data and hard disk data according to a preset configuration ratio; The master server and / or slave server pre-load the memory data into their respective memory spaces.

6. The data processing method according to claim 1, characterized in that: After configuring the query interface of the source data, the following steps are included: Obtaining a user request to be executed, wherein the user request carries user information, and the user information is used to retrieve at least one piece of user data corresponding thereto; Performing hash calculation on the user information according to the hash function to generate a query address; calling at least one piece of user data corresponding to the user information according to the query address; The at least one piece of user data is transmitted back to the user terminal that sent the user request.

7. A data processing device, characterized in that: include: An acquisition module, used for acquiring source data to be processed, wherein the source data is composed of multiple pieces of user data; A processing module, configured to set a server storing the source data as a master server of a preset distributed system, and set the remaining servers of the distributed system as slave servers of the master server, wherein the master server synchronizes the source data to the slave servers by replication, wherein the distributed system includes a plurality of sentinel nodes, and the plurality of sentinel nodes are connected to the master server and the plurality of slave servers; A storage module, used for the master server and / or the slave server to perform hash calculations on the user data one by one according to a preset hash function, and store the corresponding user data in a storage location pointed to by the hash calculation result; A configuration module, used to configure a query interface of the source data so that an external terminal can call the source data; A first sending submodule, configured for the plurality of sentinel nodes to send network packet discovery commands to the master server and the plurality of slave servers respectively; A first confirmation submodule, used to confirm whether the main server is subjectively offline according to the reply information of the main server to the network packet exploration command; The first execution submodule is used to, when the master server is in a subjective offline state, perform joint verification through the multiple sentinel nodes to determine whether the master server is objectively offline, and when the master server is in an objective offline state, select a new master server from the multiple slave servers.

8. A computer device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the data processing method according to any one of claims 1 to 6.

9. A storage medium storing computer-readable instructions, characterized in that: When the computer-readable instructions are executed by one or more processors, the one or more processors are caused to perform the steps of the data processing method according to any one of claims 1 to 6.

Images