Data processing method, device, system, electronic device and storage medium

Abstract

Embodiments of the present invention provide a data processing method, apparatus, system, electronic device, and storage medium. The method includes: obtaining a data processing instruction from a front end of a first system, the data processing instruction including a data query instruction or a data storage instruction; parsing the data processing instruction to obtain an instruction parsing result, the instruction parsing result including data query information or data storage information, the data storage information including data to be stored; if the instruction parsing result includes the data storage information, storing the data to be stored in a second system according to the data storage information, the first and second systems operating independently; and if the instruction parsing result includes the data query information, obtaining data from the first or second system according to the data query information and returning the data. By integrating the resources of the first and second systems, system resource utilization can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, device, system, electronic device and storage medium. Background Art

[0002] Fastdfs is an open-source, lightweight distributed file system designed to achieve high availability and high concurrency for file operations. It is particularly well-suited for storing small and medium-sized files. Its unique lightweight, grouped storage, and peer-to-peer architecture make it particularly well-suited for file-based online services such as photo and video websites. Consequently, it is widely used by major Chinese internet companies. Typically, the storage nodes of a single Fastdfs cluster are deployed on a storage server within the enterprise's intranet. Larger enterprises with more data often require the deployment of multiple data storage systems, potentially across multiple Fastdfs clusters within the intranet or externally. Because Fastdfs is primarily responsible for file upload and download and does not support data and storage space sharing between multiple Fastdfs clusters within or outside the intranet, the system's disaster recovery capabilities are weak and the resources of various Fastdfs clusters cannot be integrated, resulting in low storage system resource utilization. Summary of the Invention

[0003] An embodiment of the present invention provides a data processing method, which can improve resource utilization of a storage system.

[0004] In a first aspect, an embodiment of the present invention provides a data processing method, comprising the following steps:

[0005] Acquire a data processing instruction from the front end of the first system, wherein the data processing instruction includes a data query instruction or a data storage instruction;

[0006] Parsing the data processing instruction to obtain an instruction parsing result, wherein the instruction parsing result includes data query information or data storage information, and the data storage information includes data to be stored;

[0007] If the instruction parsing result includes the data storage information, the data to be stored is stored in the second system according to the data storage information, and the first system and the second system operate independently and share data;

[0008] If the instruction parsing result includes the data query information, data is obtained from the first system or the second system according to the data query information and returned.

[0009] Optionally, the data storage information further includes second path information, and storing the data to be stored in the second system according to the data storage information includes:

[0010] acquiring at least one second scheduling information from the second system according to the second path information;

[0011] The data to be stored is stored in at least one storage module of the second system according to the second scheduling information, and a storage result is obtained.

[0012] Optionally, the data storage information further includes first path information. If the storage result includes failure information, the method further includes:

[0013] acquiring at least one first scheduling information from the first system according to the first path information;

[0014] The data to be stored is stored in at least one storage module of the first system according to the first scheduling information.

[0015] Optionally, the data query information includes path information and an identifier of a data source to be queried, and the path information includes the first system identifier or the second system identifier.

[0016] Optionally, if the stored result includes success information, the method further includes:

[0017] The data to be stored is synchronized between the storage modules of the second system according to a preset synchronization rule.

[0018] Optionally, the step of acquiring data from the first system or the second system according to the data query information includes:

[0019] Obtaining the path information and the identifier of the data source to be queried in the data query information;

[0020] If the path information includes the second system identifier, determining a storage address of the data to be queried in the second system according to the identifier of the data to be queried, and acquiring data in the second system according to the storage address;

[0021] If the path information includes the identifier of the first system, the data is directly acquired in the first system.

[0022] In a second aspect, an embodiment of the present invention provides a data processing device, including:

[0023] A first acquisition module is used to acquire a data processing instruction from the front end of the first system, wherein the data processing instruction includes a data query instruction or a data storage instruction;

[0024] a parsing module, configured to parse the data processing instruction to obtain an instruction parsing result, wherein the instruction parsing result includes data query information or data storage information, and the data storage information includes data to be stored;

[0025] a storage module, configured to store the data to be stored in the second system according to the data storage information if the instruction parsing result includes the data storage information, wherein the first system and the second system operate independently and share data;

[0026] The second acquisition module is configured to acquire data from the first system or the second system according to the data query information and return the data if the instruction parsing result includes the data query information.

[0027] In a third aspect, an embodiment of the present invention provides a data processing system, comprising: a first system, a second system and the data processing device, wherein the first system comprises a front-end module and a back-end module, the front-end module is used to provide a human-computer interaction interface and obtain user input instructions; the data processing device executes the data processing method, obtains the input instructions of the front-end module of the first system and parses them to determine whether to upload data to the back-end module of the first system or the second system, or to obtain data from the first system or the second system and return

[0028] In a fourth aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the data processing method provided in the embodiment of the present invention when executing the computer program.

[0029] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the data processing method provided in the embodiment of the invention are implemented.

[0030] In an embodiment of the present invention, a data processing instruction is obtained from the front end of a first system, the data processing instruction including a data query instruction or a data storage instruction; the data processing instruction is parsed to obtain an instruction parsing result, the instruction parsing result including data query information or data storage information, the data storage information including data to be stored; if the instruction parsing result includes the data storage information, the data to be stored is stored in a second system according to the data storage information, and the first and second systems operate independently; if the instruction parsing result includes the data query information, the data is obtained from the first or second system according to the data query information and returned. A data processing instruction is obtained from the front end of the first system, data storage information is parsed therefrom, and the data to be stored is stored in the independently operating second system according to the data storage information, and the second system shares the data to be stored with the first system, or data query information is parsed and data is obtained from the first or second system according to the data query information. By integrating and sharing the resources of the first and second systems, the disaster recovery capability of the system is improved while also improving the resource utilization of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 is a flow chart of a data processing method provided by an embodiment of the present invention;

[0033] Figure 2 This is a flow chart of a data storage method provided by an embodiment of the present invention;

[0034] Figure 3 is a flowchart of another data storage method provided by an embodiment of the present invention;

[0035] Figure 4 is a flow chart of a data acquisition method provided by an embodiment of the present invention;

[0036] Figure 5 is a structural diagram of a data processing device provided by an embodiment of the present invention;

[0037] Figure 6 This is a schematic structural diagram of a data storage device provided by an embodiment of the present invention;

[0038] Figure 7This is a schematic structural diagram of another data storage device provided by an embodiment of the present invention;

[0039] Figure 8 It is a structural diagram of a data acquisition device provided by an embodiment of the present invention;

[0040] Figure 9 is a structural diagram of a data processing system provided by an embodiment of the present invention;

[0041] Figure 10 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0043] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0045] First, let's introduce the relevant background information for the embodiments of this invention. Both the first and second systems in these embodiments are implemented based on a single existing Fastdfs distributed file system. The Fastdfs file system is primarily used for file storage, file synchronization, and file access (file upload and download). The system has three roles: a tracker server, a storage server, and a client. The client is used to provide system access, such as uploading and downloading files. The tracking server mainly performs scheduling and load balancing, and is responsible for managing all storage servers and all storage nodes within the storage servers. Each storage node is divided into a group and, after startup, connects to the tracking server to inform the tracking server of its group information and sends periodic heartbeat information to the tracking server. The tracking server establishes a mapping table from storage node groups to storage servers based on the storage node heartbeat information. Multiple tracking servers have a peer-to-peer relationship, making it very easy to expand tracking server services. All tracking servers receive the heartbeat information of storage nodes and generate metadata information within the tracking server to provide read and write services. The storage server mainly provides storage capacity and data backup services. It can also be organized into groups. Each group can contain multiple storage servers, and the data within the group is backed up by each other. Organizing storage space in groups facilitates application isolation, load balancing, and customization of the number of backups. If a machine in the group fails, data recovery can be resynchronized through other machines in the group, which can improve the system's exception handling capabilities.

[0046] The Fastdfs system of the first system in the embodiment of the present invention is mainly used for temporary caching of data and storage of part of the data during big data processing. The Fastdfs system of the second system is used to store data preferentially and serve as a backup for the first system. The first system and the second system are only distinguished for convenience. In fact, the positions and functions of the two systems can be interchanged.

[0047] See Figure 1 , Figure 1 is a flow chart of a data processing method provided by an embodiment of the present invention, such as Figure 1 As shown, the following steps are included:

[0048] 101. Obtain data processing instructions from the front end of the first system.

[0049] The above-mentioned data processing instructions include data query instructions or data storage instructions.

[0050] In an embodiment of the present invention, the front end of the first system can be a visual interface of the first system based on the Fastdfs file storage system, such as a GUI or a web page, or a command line provided by the first system; the data processing instructions can be obtained through the command line entry of the front end of the first system, or through the visual interface of the first system, such as a GUI or a web page, or can be obtained and converted through the front end voice recognition of the first system. The data processing instructions can include at least one data query instruction or at least one data storage instruction, and can also include other data processing instructions, such as data deletion and appending; data query instructions can be data acquisition instructions, such as file download instructions; data storage instructions can be data upload instructions, such as image data upload instructions; the data processing instructions can be in the form of a string, or in the form of other digital formats set according to pre-set rules, such as binary format instructions.

[0051] 102. Parse the data processing instruction to obtain an instruction parsing result.

[0052] The above instruction parsing result includes data query information or data storage information, and the above data storage information includes data to be stored.

[0053] In an embodiment of the present invention, the data processing instruction may be obtained through step 101 and then the instruction may be parsed to obtain a corresponding parsing result, that is, data query information or storage information including data to be stored may be obtained. For example, the data processing instruction is first formatted and parsed. If the format of the data processing instruction is a string, such as a data query instruction of the Hypertext Transfer Protocol (HTTP) type, the corresponding file download instruction can be a string formatted host / port / downLoad / system1 / Fastdfs / tracker.storage.group.2122.jpg. The instruction is parsed according to the path parsing method of the Hypertext Transfer Protocol to obtain the location information of the file to be downloaded. The location information is further segmented to obtain the data query information, which includes the data to be queried and the specific storage path of the data. For another example, the data storage instruction upload / system2 / Fastdfs / tracker.storage.group.aadoc in the string format can be string-sliced ​​using a preset parsing rule, such as extracting the fields between " / " using a regular expression, thereby obtaining the data storage information including the data to be stored, and then temporarily stored in the memory in a certain format, such as caching in a key-value pair format. For example, the upload key field can be used to store the data upload identifier, the path field can be used to store the path information, and the data field can be used to cache the data to be stored.

[0054] 103. If the instruction parsing result includes data storage information, store the data to be stored in the second system according to the data storage information.

[0055] The first system and the second system operate independently and share data.

[0056] In an embodiment of the present invention, the instruction parsing result obtained in the above step 102 can be judged to determine whether the next data processing flow is data upload or download; specifically, if the instruction parsing result obtained includes the above data storage information, for example, the parsed information includes a string upload field or other pre-set instruction identifiers (such as 0001, ###, etc.), the data upload processing flow is performed, the data to be stored is obtained from the data field in the above data storage information, and then the storage path is obtained according to the path field in the above data storage information, and the above data to be stored is preferentially stored in the corresponding position in the above second system according to the storage path.

[0057] Furthermore, in an embodiment of the present invention, the first system is mainly used to provide a front-end entry for data processing and a back-end for big data processing and data caching. It is independently deployed in an intranet or an extranet and can be accessed through the front-end of the visual WEB page or command line. The second system is independently deployed in the intranet to store data in priority and serve as a data backup for the first system. The first and second systems can be deployed on different computer server hardware and different operating systems in the form of docker images and run independently, such as windows and linux systems. The two systems can communicate through addresses in the network (such as IP addresses) to achieve data synchronization sharing and storage space sharing. By integrating the resources of the first and second systems, the disaster recovery capability of the system for data processing and the utilization of resources can be improved.

[0058] Optional, see Figure 2 , Figure 2 This is a flow chart of a data storage method provided by an embodiment of the present invention. Figure 2 As shown, the data storage information further includes second path information. The step of storing the data to be stored in the second system according to the data storage information specifically includes:

[0059] 201. Obtain at least one second scheduling information from a second system according to second path information.

[0060] In an embodiment of the present invention, the above-mentioned second path information can be used to identify the storage path information of the data to be stored, such as the system in which it is stored, the specific storage location, etc.; the above-mentioned second scheduling information includes the second system address information of the second system in the network and the access path information of the Fastdfs storage system in the second system, and the above-mentioned access path information includes the address information of the tracking server of the above-mentioned Fastdfs storage system and the address information of the storage server and group information. Specifically, first obtain the above-mentioned second path information from the path field in the above-mentioned data storage information, such as system2 / Fastdfs / tracker.storage.group, and split the second path information of the string to obtain system2, which can be used as the identifier of the above-mentioned second system, Fastdfs can be used as the identifier of the above-mentioned Fastdfs storage system, and tracker.storage.group can be used as the optional target storage path information (which can be empty); obtain the above-mentioned second system address information from the second system according to the above-mentioned second system identifier, and obtain the address of the second system in the network (such as IP address, MAC address, etc.); finally, according to the above-mentioned Fastdfs storage system identifier, the corresponding Fastdfs system running instance can be obtained from the second system, and the tracker.storage.group field is cut to obtain the address information of the above-mentioned tracking server, the address information of the storage server and the group information.

[0061] 202. Store the data to be stored in at least one storage module of the second system according to the second scheduling information, and obtain a storage result.

[0062] In an embodiment of the present invention, the target storage location of the data to be stored in the second system can be obtained based on the second scheduling information. The size of the data to be stored is then calculated, a storage module of the same size is retrieved from the storage space of the second system, and the data to be stored is written into the storage module. Specifically, the data to be stored can be obtained from the data field in the data storage information and can be an image, a video, or a Word document, and can be binary data or encoded string data. Then, according to the second system address information in the above-mentioned second scheduling information, the position of the above-mentioned second system in the internal network is located, and the running instance of the Fastdfs system is obtained from the second system, such as the process identifier of the Fastdfs system. The above-mentioned second system address information can be an intranet IP address; then, according to the address information of the tracking server, the address information of the storage server and the group information in the above-mentioned second scheduling information, the corresponding tracking server, storage server and group can be obtained from the running instance of the above-mentioned Fastdfs system, and then the above-mentioned data to be stored is transferred to the storage server through the tracking server and written into the storage space in the corresponding group, so that the above-mentioned data to be stored is written into the storage module of the second system, and success information (such as 1, success, etc.) is returned as the above-mentioned storage result; if the above-mentioned data storage fails, for example, the acquisition of the running instance of the Fastdfs system fails, or the data to be stored exceeds the storage space of the storage module, then failure information (such as 0, fai, etc.) is returned as the above-mentioned storage result. If the address information of the tracking server, the address information of the storage server, and the group information include empty information, the data to be stored can be stored in a group with a larger storage space on the storage server of the Fastdfs system running instance, thereby making full use of the storage space and improving resource utilization.

[0063] Furthermore, the second system can synchronize the data to be stored to the first system for data sharing. Specifically, the address information of the first system in the network (such as the IP address) can be saved in the second system. When data synchronization is required, the interface of the Fastdfs storage system of the first system can be obtained according to the address information of the first system. Then, the data can be transmitted to the Fastdfs storage system of the first system in the network through the interface, thereby realizing data sharing between the two systems.

[0064] Optionally, if the stored result includes success information, the data processing method further includes:

[0065] The data to be stored is synchronized between the storage modules of the second system according to a preset synchronization rule.

[0066] In an embodiment of the present invention, the storage module of the second system includes a storage node in the storage server of the Fastdfs system. Through the data storage step, the data to be stored is finally written to the storage node. A plurality of the storage nodes form a storage node group (i.e., the storage module). Different storage node groups can store data from different data sources and the name of the data source can be used as the identifier of the corresponding storage node group. When storing data, the identifier of the storage node group can be used as a prefix of the storage name of the data to be stored to distinguish different data sources. In the storage server, each storage node group has at least one backup group with equal status. When data is stored in a storage node group, the data can be stored in another backup group according to a preset synchronization rule to achieve data synchronization between storage node groups (storage modules). The preset synchronization rule can be an incremental synchronization rule, i.e., the storage server of the Fastdfs system copies the newly added data from one storage node group to the corresponding backup group in an asynchronous manner at an appropriate time (e.g., when there is no data reading or writing), thereby improving the stability of the system.

[0067] Optional, see Figure 3 , Figure 3 is a flow chart of another data storage method provided by an embodiment of the present invention, such as Figure 3 As shown, the data storage information further includes first path information. If the storage result includes failure information, the data storage method further includes:

[0068] 301. Obtain at least one first scheduling information from a first system according to first path information.

[0069] In an embodiment of the present invention, the above-mentioned first path information can be used to identify the storage path information of the data to be stored, such as the system in which it is stored, the specific storage location, etc.; the above-mentioned first scheduling information includes the first system address information of the first system in the network and the access path information of the Fastdfs storage system in the first system, and the above-mentioned access path information includes the address information of the tracking server of the above-mentioned Fastdfs storage system and the address information of the storage server and group information. Specifically, first obtain the above-mentioned first path information from the path field in the above-mentioned data storage information, such as system1 / Fastdfs / tracker.storage.group, and split the first path information of the string to obtain system1, which can be used as the identifier of the above-mentioned first system, Fastdfs can be used as the identifier of the above-mentioned Fastdfs storage system, and tracker.storage.group can be used as the optional target storage path information (which can be empty); obtain the above-mentioned first system address information from the first system according to the above-mentioned first system identifier, and obtain the address of the first system in the network (such as IP address, MAC address, etc.); finally, according to the above-mentioned Fastdfs storage system identifier, the corresponding Fastdfs system running instance can be obtained from the first system, and the tracker.storage.group field is cut to obtain the address information of the above-mentioned tracking server, the address information of the storage server and the group information.

[0070] 302. Store the data to be stored in at least one storage module of the first system according to the first scheduling information.

[0071] In an embodiment of the present invention, the target storage location of the data to be stored in the first system can be obtained based on the first scheduling information. The size of the data to be stored is then calculated, a storage module of the same size is retrieved from the storage space of the first system, and the data to be stored is written into the storage module. Specifically, the data to be stored can be obtained from the data field in the data storage information and can be an image, a video, or a Word document, and can be binary data or encoded string data. Then, according to the first system address information in the above-mentioned first scheduling information, the position of the above-mentioned first system in the internal network is located, and the running instance of the Fastdfs system is obtained from the first system, such as the process identifier of the Fastdfs system. The above-mentioned first system address information can be an intranet IP address; then, according to the address information of the tracking server, the address information of the storage server and the group information in the above-mentioned first scheduling information, the corresponding tracking server, storage server and group can be obtained from the running instance of the above-mentioned Fastdfs system, and then the above-mentioned data to be stored is transferred to the storage server through the tracking server and written into the storage space in the corresponding group, so that the above-mentioned data to be stored is written into the storage module of the first system, and success information (such as 1, success, etc.) is returned as the above-mentioned storage result; if the above-mentioned data storage fails, for example, the acquisition of the running instance of the Fastdfs system fails, or the data to be stored exceeds the storage space of the storage module, then failure information (such as 0, fai, etc.) is returned as the above-mentioned storage result. If the address information of the tracking server, the address information of the storage server, and the group information include empty information, the data to be stored can be stored in a group with a larger storage space on the storage server of the Fastdfs system running instance, thereby making full use of the storage space and improving resource utilization.

[0072] Furthermore, the first system can synchronize the data to be stored to the second system for data sharing. Specifically, the address information of the second system in the network (such as the IP address) can be saved in the first system. When data synchronization is required, the first system can communicate with the second system according to the address information of the second system and obtain the interface of the Fastdfs storage system of the second system, and then transmit the data to the Fastdfs storage system of the second system in the network through the interface, thereby realizing data sharing between the two systems.

[0073] Optionally, the data query information includes path information and an identifier of a data source to be queried, and the path information includes an identifier of the first system or an identifier of the second system.

[0074] Among them, the above-mentioned data query information is obtained by parsing the data query instruction obtained by the front-end of the above-mentioned first system. For example, in the above-mentioned step 102, the above-mentioned data query instruction host / port / downLoad / system1 / Fastdfs / tracker.storage.group.2122.jpg in the string format can be parsed according to the path parsing method of the Hypertext Transfer Protocol to obtain the above-mentioned data query information. The above-mentioned data query information can be further segmented to obtain the above-mentioned data query information. The above-mentioned data query information includes path information and the data source identifier to be queried. The above-mentioned path information includes the identifier of the above-mentioned first system (such as the system1 field in the above-mentioned data query instruction, or 0001, etc.) or the identifier of the above-mentioned second system (such as the system2 field in the above-mentioned data query instruction, or 0010, etc.). The above-mentioned data source identifier to be queried can be the identifier of the above-mentioned storage node group of the Fastdfs storage system where the data to be queried is located (such as the group.2122 field in the above-mentioned data query instruction).

[0075] 104. If the instruction parsing result includes data query information, obtain data from the first system or the second system according to the data query information and return the data.

[0076] In an embodiment of the present invention, the instruction parsing result obtained in step 102 is judged. If the instruction parsing result includes data query information, it indicates that the original data processing instruction includes a data acquisition instruction, such as a data acquisition instruction of the Hypertext Transfer Protocol (HTTP) type in a string format, host / port / downLoad / system1 / Fastdfs / tracker.storage.group.2122.jpg. This instruction can be parsed according to the path parsing method of the Hypertext Transfer Protocol, that is, the data acquisition instruction is split according to " / " to obtain the data query information, wherein the host / port fields represent the local host address and port information, and the downLoad field can be used to indicate the data acquisition identifier. The data query information includes the data to be queried and the specific storage path of the data. For example, the storage system identifier can be obtained according to the system1 field and Fastdfs, and the specific storage path and the data to be queried can be obtained according to tracker.storage.group.2122.jpg, that is, the image data of 2122.jpg is obtained according to the tracker.storage.group path in the Fastdfs storage system of the first system, and then returned to the first system front end.

[0077] Optional, see Figure 4 , Figure 4This is a flow chart of a data acquisition method provided by an embodiment of the present invention. Figure 4 As shown, the above step of acquiring data from the first system or the second system according to the data query information includes:

[0078] 401. Obtain path information and a data source identifier to be queried in data query information.

[0079] 402. If the path information includes the identifier of the second system, determine the storage address of the data to be queried in the second system according to the identifier of the data to be queried, and obtain the data in the second system according to the storage address.

[0080] 403. If the path information includes the identifier of the first system, directly obtain the data in the first system.

[0081] In an embodiment of the present invention, path information and an identifier of the data source to be queried are obtained from the above-mentioned data query information, and then it is determined whether the above-mentioned path information contains the identifier of the above-mentioned second system or the identifier of the above-mentioned first system. If the above-mentioned path information includes the identifier of the second system (that is, the system2 field in the original data query instruction, or 0010, etc.), the address information of the above-mentioned second system in the network is first obtained, and then the storage address of the data to be queried in the second system is further determined based on the above-mentioned identifier of the data to be queried (such as the group.2122 field in the above-mentioned data query instruction), that is, the storage node group identifier of the Fastdfs running instance in the above-mentioned second system. The complete data acquisition path can be obtained through the address information of the above-mentioned second system in the network and the above-mentioned storage address, and then based on the data acquisition path, the data to be acquired is read from the corresponding storage node in the above-mentioned second system.

[0082] If the above path information includes the identifier of the first system (i.e., the system1 field in the original data query instruction, or 0001, etc.), then similarly, the complete data acquisition path is obtained through the address information of the above first system in the network and the above storage address, and then based on the data acquisition path, the data to be acquired is read from the corresponding storage node in the above first system.

[0083] To sum up, data processing instructions are obtained from the front end of the first system, data storage information is parsed therefrom, and the data to be stored is stored in an independently running second system according to the above data storage information, or data query information is parsed and then data is obtained from the above-mentioned first system or the second system according to the above-mentioned data query information. By integrating the resources of the first system and the second system, the system's disaster recovery capability is improved while the system's resource utilization is improved.

[0084] It should be noted that the data processing method provided in the embodiment of the present invention can be applied to devices such as mobile phones, monitors, computers, servers, etc. that can perform data processing.

[0085] See Figure 5 , Figure 5 is a structural diagram of a data processing device provided by an embodiment of the present invention, such as Figure 5 As shown, the apparatus 500 includes:

[0086] A first acquisition module 501 is configured to acquire a data processing instruction from a front-end of a first system, wherein the data processing instruction includes a data query instruction or a data storage instruction;

[0087] A parsing module 502 is configured to parse the data processing instruction to obtain an instruction parsing result, wherein the instruction parsing result includes data query information or data storage information, and the data storage information includes data to be stored;

[0088] a storage module 503 configured to store the data to be stored in the second system according to the data storage information if the instruction parsing result includes the data storage information, and the first system and the second system operate independently and share data;

[0089] The second acquisition module 504 is configured to, if the instruction parsing result includes the data query information, acquire data from the first system or the second system according to the data query information and return the data.

[0090] Optional, such as Figure 6 As shown, Figure 6 is a structural diagram of a data storage device provided by an embodiment of the present invention. The data storage information further includes second path information. The data storage device 600 includes:

[0091] An acquisition module 601 is configured to acquire at least one second scheduling information from a second system according to the second path information;

[0092] The storage module 602 is configured to store the data to be stored in at least one storage module of the second system according to the second scheduling information, and obtain a storage result.

[0093] Optional, such as Figure 7 As shown, Figure 7 is a structural diagram of another data storage device provided by an embodiment of the present invention. The data storage information further includes first path information. If the storage result includes failure information, the data storage device 700 further includes:

[0094] An acquisition module 701 is configured to acquire at least one first scheduling information from a first system according to the first path information;

[0095] The storage module 702 is configured to store the data to be stored in at least one storage module of the first system according to the first scheduling information.

[0096] Optional, such as Figure 8 As shown, Figure 8 8 is a schematic structural diagram of a data acquisition device provided by an embodiment of the present invention. The data acquisition device 800 includes:

[0097] The first acquisition module 801 is used to obtain the path information and the identifier of the data source to be queried in the data query information;

[0098] A second acquisition module 802 is configured to determine a storage address of the data to be queried in the second system according to the identifier of the data to be queried, if the path information includes an identifier of the second system, and acquire the data in the second system according to the storage address;

[0099] The third acquisition module 803 is configured to directly acquire data from the first system if the path information includes the identifier of the first system.

[0100] It should be noted that the data processing device provided in the embodiment of the present invention can be applied to devices such as mobile phones, monitors, computers, servers, etc. that can perform data processing.

[0101] The data processing device provided in the embodiment of the present invention can implement each process implemented by the data processing method in the above method embodiment and can achieve the same beneficial effects. To avoid repetition, it will not be described here.

[0102] See also Figure 9 , Figure 9 is a structural diagram of a data processing system provided by an embodiment of the present invention, such as Figure 9 As shown, the above-mentioned data processing system 900 includes: a first system 901, a second system 902 and the above-mentioned data processing device 500, wherein the first system 901 includes a front-end module 9011 and a back-end module 9012, and the above-mentioned front-end module 9011 is used to provide a human-computer interaction interface and obtain user input instructions; the above-mentioned data processing device 500 executes the above-mentioned data processing method, obtains the input instructions of the above-mentioned first system front-end module 9011 and parses it to determine whether to upload the data to the back-end module 9012 of the above-mentioned first system 901 or the above-mentioned second system 902, or to obtain data from the above-mentioned first system 901 or the second system 902 and return it.

[0103] Both the first system 901 and the second system 902 are built based on the Fastdfs storage system and can be independently deployed within the same intranet or on different networks, sharing data over the network. The first system 901 can be the first system in the data processing method described above, primarily used for big data processing, temporary data caching, and partial data storage. The second system 902 can be the second system in the data processing method described above, used for prioritizing data storage and serving as a backup for the first system 901. By integrating the resources of the first system 901 and the second system 902, the resource utilization of the system 900 is improved while simultaneously enhancing the disaster recovery capability of the system 900.

[0104] See also Figure 10 , Figure 10 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention, such as Figure 10 As shown, it includes: a memory 1002, a processor 1001, and a computer program stored in the memory 1002 and executable on the processor 1001, wherein:

[0105] The processor 1001 is configured to call the computer program stored in the memory 1002 and execute the following steps:

[0106] Acquire a data processing instruction from the front end of the first system, wherein the data processing instruction includes a data query instruction or a data storage instruction;

[0107] Parsing the data processing instruction to obtain an instruction parsing result, wherein the instruction parsing result includes data query information or data storage information, and the data storage information includes data to be stored;

[0108] If the instruction parsing result includes the data storage information, the data to be stored is stored in the second system according to the data storage information, and the first system and the second system operate independently and share data;

[0109] If the instruction parsing result includes the data query information, data is obtained from the first system or the second system according to the data query information and returned.

[0110] Optionally, the data storage information further includes second path information, and the storing of the data to be stored in the second system according to the data storage information performed by the processor 1001 includes:

[0111] acquiring at least one second scheduling information from the second system according to the second path information;

[0112] The data to be stored is stored in at least one storage module of the second system according to the second scheduling information, and a storage result is obtained.

[0113] Optionally, the data storage information further includes first path information. If the storage result includes failure information, the method executed by the processor 1001 further includes:

[0114] acquiring at least one first scheduling information from the first system according to the first path information;

[0115] The data to be stored is stored in at least one storage module of the first system according to the first scheduling information.

[0116] Optionally, if the stored result includes success information, the method executed by the processor 1001 further includes:

[0117] The data to be stored is synchronized between the storage modules of the second system according to a preset synchronization rule.

[0118] Optionally, the step of acquiring data from the first system or the second system according to the data query information, performed by the processor 1001, includes:

[0119] Obtaining the path information and the identifier of the data source to be queried in the data query information;

[0120] If the path information includes the second system identifier, determining a storage address of the data to be queried in the second system according to the identifier of the data to be queried, and acquiring data in the second system according to the storage address;

[0121] If the path information includes the identifier of the first system, the data is directly acquired in the first system.

[0122] It should be noted that the above-mentioned electronic devices can be applied to mobile phones, monitors, computers, servers and other devices that can perform data processing.

[0123] The electronic device provided by the embodiment of the present invention can implement each process implemented by the data processing method in the above method embodiment and can achieve the same beneficial effects. To avoid repetition, it will not be described here.

[0124] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the data processing method provided by the embodiment of the present invention are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0125] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0126] The above disclosure is merely a preferred embodiment of the present invention and certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A data processing method, characterized in that: The following steps are involved: Acquire a data processing instruction from the front end of the first system, wherein the data processing instruction includes a data query instruction or a data storage instruction; Parsing the data processing instruction to obtain an instruction parsing result, wherein the instruction parsing result includes data query information or data storage information, and the data storage information includes data to be stored; If the instruction parsing result includes the data storage information, the data to be stored is stored in the second system according to the data storage information. The first system and the second system operate independently and share data. The first system is used to provide a front-end entry for data processing and a back-end for big data processing and data caching. The first system is independently deployed on the external network, and the second system is independently deployed on the internal network. It is used to store data first and serve as a data backup for the first system. The Fastdfs system of the first system is used for temporary caching of data and storage of part of the data during big data processing. The Fastdfs system of the second system is used to store data first and serve as a backup for the first system. The data storage information further includes second path information, and storing the data to be stored in the second system according to the data storage information includes: Acquire at least one second scheduling information from the second system according to the second path information; the second scheduling information includes second system address information of the second system in the network and access path information of the second system Fastdfs system; According to the second scheduling information, the data to be stored is stored in at least one storage module of the second system, and a storage result is obtained; according to the address information of the tracking server, the address information of the storage server and the group information in the second scheduling information, the corresponding tracking server, storage server and group are obtained from the running instance of the Fastdfs system in the second system, and then the data to be stored is transferred to the storage server through the tracking server and written into the storage space in the corresponding group, so that the data to be stored is written into the storage module of the second system, and a success message is returned as a storage result; if the acquisition of the running instance of the Fastdfs system in the second system fails, or the data to be stored exceeds the storage space of the storage module of the second system, a failure message is returned as a storage result; if the address information of the tracking server, the address information of the storage server and the group information include empty information, the data to be stored is stored in the group with larger storage space on the storage server of the running instance of the Fastdfs system in the second system; If the storage result includes success information, the data to be stored is synchronized between the storage modules of the second system according to a preset synchronization rule; the preset synchronization rule is an incremental synchronization rule, specifically, when there is no data reading or writing, the newly added data is copied from one storage node group to the corresponding backup group in an asynchronous manner; The data storage information also includes first path information. If the storage result includes failure information, at least one first scheduling information is obtained from the first system according to the first path information; the first scheduling information includes first system address information of the first system in the network and access path information of the Fastdfs system in the first system, and the access path information includes address information of the tracking server and address information of the storage server of the Fastdfs system, as well as group information; According to the first scheduling information, the data to be stored is stored in at least one storage module of the first system; according to the address information of the tracking server, the address information of the storage server and the group information in the first scheduling information, the corresponding tracking server, storage server and group are obtained from the running instance of the Fastdfs system in the first system, and then the data to be stored is transferred to the storage server through the tracking server and written into the storage space in the corresponding group, so that the data to be stored is written into the storage module of the first system, and a success message is returned as a storage result; if the acquisition of the running instance of the Fastdfs system in the first system fails, or the data to be stored exceeds the storage space of the storage module of the first system, a failure message is returned as a storage result; if the address information of the tracking server, the address information of the storage server and the group information include empty information, the data to be stored is stored in the group with larger storage space on the storage server of the running instance of the Fastdfs system in the first system; If the instruction parsing result includes the data query information, data is obtained from the first system or the second system according to the data query information and returned.

2. The method according to claim 1, wherein The data query information includes path information and an identifier of a data source to be queried, and the path information includes an identifier of the first system or an identifier of the second system.

3. The method according to claim 2, wherein The step of acquiring data from the first system or the second system according to the data query information includes: Obtaining the path information and the identifier of the data source to be queried in the data query information; If the path information includes the identifier of the second system, determining the storage address of the data to be queried in the second system according to the identifier of the data source to be queried, and acquiring the data in the second system according to the storage address; If the path information includes the identifier of the first system, the data is directly acquired in the first system.

4. A data processing device, characterized in that: include: A first acquisition module is used to acquire a data processing instruction from the front end of the first system, wherein the data processing instruction includes a data query instruction or a data storage instruction; a parsing module, configured to parse the data processing instruction to obtain an instruction parsing result, wherein the instruction parsing result includes data query information or data storage information, and the data storage information includes data to be stored; A storage module, configured to store the data to be stored in a second system according to the data storage information if the instruction parsing result includes the data storage information, wherein the first system and the second system operate independently and share data; the first system is configured to provide a front-end entry for data processing and a back-end for big data processing and data caching, the first system being independently deployed on an external network, and the second system being independently deployed on an internal network, for prioritizing data storage and serving as a data backup for the first system; the Fastdfs system of the first system is configured to temporarily cache data and store partial data during big data processing, and the Fastdfs system of the second system is configured to prioritize data storage and serve as a backup for the first system; The data storage information also includes second path information, and storing the data to be stored in the second system according to the data storage information includes: obtaining at least one second scheduling information from the second system according to the second path information; the second scheduling information includes the second system address information of the second system in the network and the access path information of the second system Fastdfs system; storing the data to be stored in at least one storage module of the second system according to the second scheduling information, and obtaining the storage result; obtaining the corresponding tracking server, storage server and group from the running instance of the Fastdfs system in the second system according to the address information of the tracking server, the address information of the storage server and the group information in the second scheduling information, and then transferring the data to be stored to the storage server through the tracking server and writing it into the storage space in the corresponding group, thereby writing the data to be stored into the storage module of the second system, and returning success information as the storage result; If the acquisition of the running instance of the Fastdfs system in the second system fails, or the data to be stored exceeds the storage space of the storage module of the second system, the failure information is returned as the storage result; if the address information of the tracking server, the address information of the storage server and the group information include empty information, the data to be stored is stored in the group with larger storage space on the storage server of the running instance of the Fastdfs system in the second system; if the storage result includes success information, the data to be stored is synchronized between the storage modules of the second system according to the preset synchronization rules; the preset synchronization rule is an incremental synchronization rule, specifically, when there is no data reading or writing, the newly added data is copied from one storage node group to the corresponding backup group in an asynchronous manner; the data storage information also includes first path information. If the storage result includes failure information, the data is copied from the storage node group according to the first path information. At least one first scheduling information is obtained in the first system; the first scheduling information includes the first system address information of the first system in the network and the access path information of the Fastdfs system in the first system, and the access path information includes the address information of the tracking server and the address information and group information of the storage server of the above-mentioned Fastdfs system; according to the first scheduling information, the data to be stored is stored in at least one storage module of the first system; according to the address information of the tracking server, the address information of the storage server and the group information in the first scheduling information, the corresponding tracking server, storage server and group are obtained from the running instance of the Fastdfs system in the first system, and then the data to be stored is transferred to the storage server through the tracking server and written into the storage space in the corresponding group, so that the data to be stored is written into the storage module of the first system, and a success message is returned as a storage result;If the acquisition of the running instance of the Fastdfs system in the first system fails, or the data to be stored exceeds the storage space of the storage module of the first system, a failure message is returned as the storage result; if the address information of the tracking server, the address information of the storage server, and the group information include null information, the data to be stored is stored in a group with a larger storage space on the storage server of the running instance of the Fastdfs system in the first system; The second acquisition module is configured to acquire data from the first system or the second system according to the data query information and return the data if the instruction parsing result includes the data query information.

5. A data processing system, characterized in that: include: A first system, a second system and a data processing device as described in claim 4, wherein the first system includes a front-end module and a back-end module, the front-end module is used to provide a human-computer interaction interface and obtain user input instructions; the data processing device executes the data processing method as described in any one of claims 1 to 3, obtains the input instructions of the front-end module of the first system, parses them, and then determines whether to upload data to the back-end module of the first system or the second system, or obtain data from the first system or the second system and return it.

6. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data processing method according to any one of claims 1 to 3 when executing the computer program.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the data processing method according to any one of claims 1 to 3.

Images