Data processing method and device, equipment, storage medium and program product

Abstract

The invention discloses a data processing method and device, equipment, a storage medium and a program product, and relates to the technical field of data processing.The data storage system can meet the requirement that service requests of different service types are processed by adopting customized data processing logic through multiple service processing modules, and meanwhile, the data processing efficiency is improved. Target data can be managed in a customized manner for business requests of different business types through a plurality of storage devices; by adopting the data storage system, different service systems do not need to correspondingly develop data processing functions for different service requests, and data processing can be completed through the same service processing module for common service requests existing in different service systems, so that the data processing efficiency is improved. The service processing efficiency can be improved while personalized processing of different service requests of different service systems is met.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, apparatus, device, storage medium and program product. Background Art

[0002] At present, in order to meet different business requests of business systems, corresponding data processing functions are usually developed for the business requests of different business systems, and a fixed data storage system is developed to facilitate data operations of the business systems.

[0003] That is, the business system not only needs to implement business processing functions, but also needs to implement data processing functions; and, since different business systems also have common business requests, if the above method is adopted, the business processing efficiency of the business system will be reduced. Summary of the invention

[0004] The main purpose of this application is to provide a data processing method, device, equipment, storage medium and program product, aiming to solve the technical problem of low business processing efficiency of business systems.

[0005] To achieve the above purpose, the present application proposes a data processing method, which includes:

[0006] Applied to a data storage system, the data storage system includes multiple business processing modules and a storage management module, the storage management module is used to manage multiple storage devices, and the method includes:

[0007] In response to the service request, determining a target service processing module based on the target service type to which the service request belongs, wherein each of the service processing modules corresponds to a different data processing logic;

[0008] The target data corresponding to the business request is processed by the target business processing module to obtain a processing result, wherein the target data is data obtained from a target storage device corresponding to the target business type, or is data that needs to be stored in the target storage device.

[0009] In one embodiment, the step of processing the target data corresponding to the service request by the target service processing module to obtain a processing result includes at least one of the following:

[0010] Generate a unique identifier of the target data to be stored in the target storage device based on the target business type through the target business processing module, process the target data to obtain processed data, and store the processed data in the storage device corresponding to the unique identifier through the storage management module to obtain a processing result;

[0011] The target business processing module obtains target data from the target storage device based on the unique identifier carried in the business request, processes the target data, and obtains a processing result.

[0012] In one embodiment, the business request further carries a business requirement, the business requirement includes storing target data, and the target data includes unstructured data;

[0013] The step of processing the target data to obtain processed data comprises:

[0014] Convert the unstructured data into a target data structure to obtain converted data;

[0015] Based on the business requirements of storing the target data, generating target metadata corresponding to the unstructured data;

[0016] The unique identifier, the converted data and the target metadata are packaged to obtain processed data.

[0017] In one embodiment, the step of storing the processed data in a storage device corresponding to the unique identifier through the storage management module to obtain a processing result includes:

[0018] Determining, by the storage management module, a target storage device based on a unique identifier in the processed data;

[0019] The converted data and the target metadata are respectively stored in a storage device corresponding to the unique identifier.

[0020] In one embodiment, the storage management module is provided with a plurality of device adapter modules, and the storage management module is respectively connected to the storage devices in communication via the plurality of device adapter modules;

[0021] The step of storing the processed data in a storage device corresponding to the unique identifier through the storage management module to obtain a processing result includes:

[0022] Determine, by the storage management module, a target device adapter module based on the unique identifier;

[0023] The target device adapter module stores the processed data in the target storage device based on the preset device connection information of the target storage device to obtain a processing result.

[0024] In one embodiment, before the step of processing the target data corresponding to the service request by the target service processing module to obtain a processing result, at least one of the following items is further included:

[0025] When the storage management module detects that the storage device has changed, the mapping relationship between the unique identifier and the device adapter module is changed, and the change of the storage device includes at least one of a device failure and a device offline;

[0026] Based on the target service level to which the service request belongs, the mapping relationship between the unique identifier and the device adaptation module is changed.

[0027] In addition, to achieve the above-mentioned purpose, the present application also proposes a data processing device, the data processing device comprising:

[0028] A gateway module, configured to respond to a service request and determine a target service processing module based on a target service type to which the service request belongs, wherein each of the service processing modules corresponds to a different data processing logic;

[0029] The business processing module is used to process the target data corresponding to the business request through the target business processing module to obtain a processing result, wherein the target data is data obtained from a target storage device corresponding to the target business type, or data that needs to be stored in the target storage device.

[0030] In addition, to achieve the above objectives, the present application also proposes a data processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method described above.

[0031] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable and writable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the data processing method described above are implemented.

[0032] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0033] One or more technical solutions proposed in this application have at least the following technical effects:

[0034] The data storage system of the present application includes multiple business processing modules and a storage management module, and the storage management module is used to manage multiple storage devices; the data storage system responds to a business request and determines a target business processing module from multiple business processing modules based on the target business type to which the business request belongs. Since each of the business processing modules corresponds to a different data processing logic, the target business processing module processes the target data corresponding to the business request obtained from the target storage device corresponding to the target business type, or the target data corresponding to the business request that needs to be stored in the target storage device based on its data processing logic, to obtain a processing result.

[0035] It can be understood that the data storage system of the present application can meet the needs of business requests of different business types by adopting customized data processing logic for processing through multiple business processing modules. At the same time, it can also manage the target data in a customized manner for business requests of different business types through multiple storage devices. By adopting the data storage system of the present application, there is no need for different business systems to develop data processing functions corresponding to different business requests, and data processing can be completed through the same business processing module for common business requests existing in different business systems. Therefore, the present application can improve business processing efficiency while meeting the personalized processing of different business requests of different business systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0038] Figure 1 A flowchart of the first embodiment of the data processing method of the present application is provided;

[0039] Figure 2 A schematic diagram of a first scenario provided for Embodiment 1 of the data processing method of this application;

[0040] Figure 3 A schematic diagram of a second scenario provided in Example 1 of the data processing method of this application;

[0041] Figure 4 A schematic diagram of a third scenario provided in Embodiment 2 of the data processing method of this application;

[0042] Figure 5A schematic diagram of the fourth scenario provided in Embodiment 2 of the data processing method of this application;

[0043] Figure 6 A schematic diagram of the fifth scenario provided in Embodiment 3 of the data processing method of this application;

[0044] Figure 7 This is a schematic diagram of the module structure of the data processing device according to an embodiment of the present application.

[0045] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0046] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0047] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0048] Based on this, the present application embodiment provides a data processing method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the data processing method of the present application.

[0049] In this embodiment, the data processing method includes steps S10 to S20:

[0050] Step S10, in response to the service request, determining a target service processing module based on the target service type to which the service request belongs, wherein each of the service processing modules corresponds to a different data processing logic;

[0051] It should be noted that the execution subject of this embodiment is a data storage system, or may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a data processing device, etc. The following takes the data storage system as an example to illustrate this embodiment and the following embodiments.

[0052] Specifically, refer to Figure 2The data storage system includes multiple business processing modules and a storage management module, and the storage management module is used to manage multiple storage devices; the data storage system also includes a user interaction module and a gateway module, wherein the user interaction module can be a client or a web page that interacts with the user, and the gateway module is used to process the data stream; wherein the storage device is the underlying device that actually stores the file, for example, a hard disk, an NFS (Network File System, Network File System Protocol) storage device, a VSAN (vSphere Virtual SAN, storage virtualization technology), an object storage, etc., and may also directly use a database for storage, and the storage management module is used to manage the above storage devices.

[0053] In order to meet the different business requests of business systems in the related technologies, corresponding data processing functions are usually developed for the business requests of different business systems, and a fixed data storage system is developed to facilitate data operations of the business systems, which leads to a decrease in the business processing efficiency of the business systems. The present embodiment aims to meet the personalized processing of different business requests of different business systems through the above-mentioned data storage system, while improving the business processing efficiency.

[0054] Specifically, different business requests of a business system include different data management requirements. For example, the business system needs to store data in a data storage system. After a preset time period, the data storage system does not need to provide a modification function, but needs to provide a function of viewing at any time; the business system needs to store data in a data storage system, and requires the data storage system to support frequent modifications; the business system requires the data storage system to provide a regular cleanup function, or to regularly generate static links for sharing, etc.

[0055] Therefore, the data storage system provides different processing methods for different types of business requests, without the need to develop data processing functions in the business system.

[0056] Specifically, refer to Figure 3 The user can trigger a business request through the user interaction module of the data storage system, or can send a business request to the user interaction module of the data storage system through the business system; the business request needs to carry a business type identifier, for example, the business type identifier can be a data storage type, a data modification type, a data reading type, etc.

[0057] It can be understood that different business requests correspond to different business processing modules. A business processing module is a data processing logic designed specifically for a certain business request. The user interaction module responds to the business request and determines the target business processing module based on the target business type of the business request, thereby realizing customized processing of different business requests.

[0058] Specifically, the user interaction module may respond to the service request and send the service request to the gateway module, so that the gateway module determines the target service processing module based on the target service type to which the service request belongs, and sends the service request to the target service processing module.

[0059] In the specific implementation process, each business processing module can be deployed as an independent distributed instance cluster (each business processing module can be expanded and optimized according to its own needs without affecting other business processing modules), and a unified service interface can be provided to the outside through the microservice architecture. The gateway module serves as the entrance to all external requests, and can receive business requests sent by the user interaction module. According to the target business type, the business request is forwarded to the corresponding service instance of the target business processing module. This approach improves the flexibility and maintainability of the system, while also enhancing the scalability and fault isolation capabilities of the system.

[0060] It should be noted that the data storage system also includes a main control monitoring module, which stores the association relationship between the business type and the business processing module. When the gateway module receives a business request sent by the user interaction module, it can obtain the association relationship between the business type and the business processing module from the main control monitoring module, thereby determining the target business processing module.

[0061] Step S20, processing the target data corresponding to the business request through the target business processing module to obtain a processing result, wherein the target data is data obtained from a target storage device corresponding to the target business type, or data that needs to be stored in the target storage device.

[0062] Since the storage management module manages multiple storage devices, the target data corresponding to the business request may be all already stored in the target storage device corresponding to the target business type, or may be all that needs to be stored in the target storage device corresponding to the target business type; the target data corresponding to the business request may also be partially already stored in the target storage device corresponding to the target business type, and partially that needs to be stored in the target storage device corresponding to the target business type.

[0063] Therefore, in the process of the target business processing module processing the target data corresponding to the business request, the target data can be obtained from the target storage device corresponding to the target business type, and then the target data can be processed to obtain the processing result, or the target data can be stored in the target storage device through processing to obtain the processing result.

[0064] Since data is interacted with the target storage device during the process of processing business requests, that is, dedicated storage devices are provided for different business types, there is no need to develop fixed data storage systems for different business systems, thus reducing resource waste.

[0065] In one implementation, the step of processing the target data corresponding to the service request by the target service processing module to obtain a processing result includes at least one of the following:

[0066] First, through the target business processing module, based on the target business type, a unique identifier of the target data that needs to be stored in the target storage device is generated, and the target data is processed to obtain processed data, and the processed data is stored in the storage device corresponding to the unique identifier through the storage management module to obtain a processing result.

[0067] It can be understood that if the business request is for data storage, the business request can be sent to the target business processing module, and the target business processing logic is executed by the target business processing module: based on the target business type, a unique identifier of the target data that needs to be stored in the target storage device is generated. The unique identifier can be a key value, or a serial number, etc.

[0068] Since different business types correspond to different storage devices, the unique identifier generated based on the target business type of the business request corresponds to the target storage device; therefore, the business processing module can send the correspondence between the unique identifier and the target storage device to the main control monitoring module.

[0069] Furthermore, the target service processing logic further includes: processing the target data to obtain processed data, sending the processed data to the storage management module, referring to Figure 3 .

[0070] It can be understood that after receiving the processed data, the storage management module can store the processed data in the target storage device to obtain the processing result, for example, the processing result is successful storage or completion of the business request, etc. At the same time, the unique identifier of the target data and the processing result can be fed back to the user; specifically, after obtaining the processing result, the storage management module can feed back the processing result to the business system or client through the user interaction module.

[0071] Second, through the target service processing module, based on the unique identifier carried in the service request, the target data is obtained from the target storage device, and the target data is processed to obtain a processing result.

[0072] Based on the above, when the target data needs to be read, the corresponding relationship can be obtained from the main control monitoring module based on the unique identifier to determine the target storage device. Figure 3 .

[0073] It should be noted that if the business request is a request other than data storage, the user can add a unique identifier of the data to be processed in the business request. The target business processing module can obtain the target data from the target storage device based on the unique identifier carried by the business request, thereby processing the target data and obtaining the processing result.

[0074] For example, when the business request is for data reading, the target data can be obtained from the target storage device based on the unique identifier of the data to be read, and the target data can be converted into the data format required by the user to obtain the processing result; when the business request is for data modification, the target data can be obtained from the target storage device based on the unique identifier of the data to be read, and the target data can be modified, and then the modified data can be stored in the target storage device to obtain the processing result, etc.

[0075] In this embodiment, the data storage system can meet the needs of business requests of different business types by using customized data processing logic for processing through multiple business processing modules. At the same time, it can also manage the target data in a customized manner for business requests of different business types through multiple storage devices. By using the data storage system of the present application, there is no need for different business systems to develop data processing functions corresponding to different business requests, and the data processing can be completed through the same business processing module for common business requests existing in different business systems. Therefore, the present application can improve business processing efficiency while meeting the personalized processing of different business requests of different business systems. Moreover, the target data corresponding to the business requests of different business types interact with the target storage devices respectively without affecting each other, thereby improving data processing reliability.

[0076] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description, and will not be described in detail later. On this basis, the specific implementation method of processing the target data to obtain the processed data can be:

[0077] The unstructured data is converted into a target data structure to obtain converted data; based on the business requirements for storing the target data, target metadata corresponding to the unstructured data is generated; the unique identifier, the converted data and the target metadata are packaged to obtain processed data.

[0078] It should be noted that the business request also carries business requirements, which include storing target data, and the target data includes unstructured data; specifically, unstructured data refers to data that does not have a predefined data model or format, such as text data, images or video data, audio data or web page data in different formats.

[0079] Since the formats of target data corresponding to different business requests are different, the current business system needs to develop a data format conversion function specifically to process unstructured data, which reduces the business processing efficiency of the business system.

[0080] The target business processing module of this embodiment executes the target business processing logic, converts the unstructured data into a target data structure to obtain converted data; generates target metadata corresponding to the unstructured data based on the business requirements for storing the target data; and packages the unique identifier, converted data and target metadata to obtain processed data.

[0081] That is, based on the above method, customized logical processing of unstructured data can be realized, and different types of unstructured data can be processed, and metadata can be generated based on business needs (for example, replacing imaging services, adding imaging services, etc.), and the unique identifier, converted data and target metadata are packaged to obtain processed data.

[0082] At present, relational databases or ES (Elastic Search, a distributed search and analysis engine) are usually used to store metadata, and the metadata and the actually stored unstructured data are mapped to ensure that users can obtain the unstructured data stored in the system through the metadata. In this storage method, although metadata and unstructured data correspond one-to-one, as the amount of data grows, the volume of metadata will become very large, and the database architecture cannot support such a large amount of data (because it is necessary to maintain additional data such as indexes that occupy the disk). Although it can also be processed in separate libraries and tables, the maintenance cost is still very high when redundancy or migration is required.

[0083] Therefore, due to the need to control the amount of data, the size of the metadata itself cannot be too large, which limits the scalability of the metadata field. Generally, unstructured data exists in the form of binary fields or files, and the lengths vary greatly. Metadata is an important feature for describing unstructured data. Metadata can not only reduce the overhead of converting binary into readable information, but also obtain the basic information required by users. Metadata can be used to determine whether the content of unstructured data is what users need. However, the user information that metadata can actually contain is relatively scarce, with only relatively common fields and limited length. Therefore, the number and length of metadata fields cannot be adjusted according to user needs, nor can connections be established between different unstructured data. Therefore, the storage system in the relevant technology only supports saving brief fields such as file names and creation and modification times, and cannot support scenarios that require storage of custom metadata.

[0084] And, refer to Figure 4 Currently, metadata is usually stored in a metadata database. Since multiple unstructured data may be of different versions, or may be main items and attachments, or a business request may be associated with metadata of several or dozens of unstructured data, the user's database resources will be wasted during the data storage process. When sharing data, it is necessary to obtain the metadata from the metadata database and then obtain the original data associated with the metadata. Therefore, this data storage method is not convenient for data sharing.

[0085] In order to solve the above problem, the specific implementation method of storing the processed data in the storage device corresponding to the unique identifier through the storage management module to obtain the processing result may be:

[0086] By means of the storage management module, based on the unique identifier in the processed data, a target storage device is determined; the converted data and the target metadata are respectively stored in the storage device corresponding to the unique identifier; referring to Figure 3 .

[0087] Specifically, the storage management module can determine the target storage device based on the unique identifier in the processed data, and store the converted data and the target metadata in the storage device corresponding to the unique identifier, that is, this embodiment does not adopt a traditional metadata database, because the use of a metadata database for data storage requires separate storage of metadata for unstructured data, and the metadata is separated from the unstructured data itself, which is not conducive to data migration or synchronization; and the metadata field is fixed and cannot support business expansion.

[0088] This embodiment manages metadata in the same way as processed data, that is, metadata and processed data are stored in the same way to ensure that the storage locations of metadata and processed data are the same, which is convenient for data maintenance and matching, and also convenient for expansion of metadata storage space; at the same time, processed data and metadata can also be stored in different storage devices in the form of files respectively, and associated based on a unique identifier. When a business request involves data migration, the processed data file and metadata file can be located based on the unique identifier and migrated synchronously.

[0089] Reference Figure 5 The unstructured data file with the key 10001metadata stores metadata information of five unstructured data files, namely, 10001data-10005data. Specifically, the metadata information includes version information of each unstructured data, file name, file type, latest version, deletion mark, etc. of each unstructured data.

[0090] In this embodiment, the metadata structure of different businesses can also be customized to a certain extent according to business needs, so that unstructured data can be bound to necessary business information without the need for the business system to perform additional relationship maintenance of the metadata database. Even if the data of the business system is lost, some key business information can be restored with the help of metadata. On the basis of supporting distributed processing of unstructured data, personalized storage resources can be provided for different business requests.

[0091] Based on the first and second embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description, and will not be described in detail later. On this basis, the specific implementation method of storing the processed data in the storage device corresponding to the unique identifier through the storage management module to obtain the processing result can be:

[0092] Through the storage management module, based on the unique identifier, a target device adapter module is determined; through the target device adapter module, based on the preset device connection information of the target storage device, the processed data is stored in the target storage device to obtain a processing result.

[0093] Although a single piece of unstructured data is only a few KB to a few MB in size, the number of files that need to be processed each week may be in the hundreds of millions. Therefore, using fixed storage devices for data storage may result in insufficient storage space and an inability to properly manage unstructured data.

[0094] At present, in order to solve the above problems, the following methods are usually used for data storage: first, use the database to save binary Blob (Binary Large Object, usually used to store unstructured or semi-structured data) data. This method has a large overhead and is not conducive to data migration, replication, and transfer; second, use distributed cloud storage resources (for example, object storage, HBase (a data storage system built on Hadoop), etc.) to save file data, and use the database to manage resource keys.

[0095] The above method cannot provide different data management methods for different business types and has poor flexibility.

[0096] At the same time, the supplier's own management tools or SDK (Software Development Kit) are usually used to operate the underlying storage resources. However, this method can only read and write a single type of underlying storage resources. In actual business scenarios, due to supplier changes, equipment updates and upgrades, new equipment replacements, etc., as the amount of data grows, a single device is often not enough to meet the demand, resulting in the need to manage multiple devices at the same time; at this time, a single storage management module cannot cope with the complex storage management needs of multiple types and multiple devices.

[0097] In order to solve the above problems, the storage management module described in this embodiment is provided with multiple device adapter modules, and the storage management module is communicated and connected with each of the storage devices through multiple device adapter modules respectively; specifically, various different storage devices in this embodiment are abstracted into a unified interface, which is implemented through an adapter module, so that the storage management module can manage different types of storage resources in a consistent manner; these device adapter modules together constitute a distributed virtual storage device that is theoretically infinitely expandable and partially pluggable.

[0098] In actual applications, the above method can be used to abstract devices such as NFS, object storage, and local disks. For the storage management module, the storage management module can realize data access, data synchronization, and data migration between different types of devices based on the connection information of each storage device. Figure 6 .

[0099] Specifically, the storage devices managed by the storage management module can be various storage devices managed internally by the data storage system, or they can be storage devices provided by users. The storage management module can manage them through the connection information of the storage devices; even if the data storage system currently does not support this device type, it only requires lightweight adaptation and deployment of a device adapter module instance to achieve management of this type of device.

[0100] Therefore, the storage management module described in this embodiment determines the target device adapter module based on the unique identifier; through the target device adapter module, the storage path is determined based on the preset device connection information of the target storage device, and the processed data can be stored in the target storage device to obtain the processing result.

[0101] This method greatly simplifies the cost of adding, switching, and removing storage devices, allowing the data storage system to plug and unplug some storage devices at any time due to business needs without affecting other business requests.

[0102] It should be noted that the main control monitoring module of this embodiment can obtain the operating data of each storage device through the storage management module in real time, so as to regularly monitor the usage of each storage device (including operating status, resource usage, etc.) to ensure the normal operation of the overall data storage system.

[0103] In this embodiment, before the step of processing the target data corresponding to the service request by the target service processing module to obtain a processing result, there is at least one of the following feasible implementation methods:

[0104] First, when the storage management module detects that the storage device has changed, the mapping relationship between the unique identifier and the device adapter module is changed, and the change of the storage device includes at least one of a device failure and a device offline.

[0105] It should be noted that in the actual production environment, there will be situations where multiple underlying devices of different models or different architectures are used in collaboration, and there will be different business requests that require autonomous management of storage resources, rather than being fully managed by the storage management module (for example, when the business system has business needs for data mining or regular review, it may be necessary to take the entire storage device offline. Therefore, in this case, the business system needs to be responsible for processing the data); if the data storage system itself does not support customized expansion of the underlying storage resources and can only manage its own adapted resources, it will not be able to support the above-mentioned scenario of autonomous management of storage resources, nor can it achieve device-level data isolation based on business type; if it is necessary to extract unstructured data for all businesses, data can only be searched based on keys or directories.

[0106] The storage management module of this embodiment is based on a hot-swappable device adapter module, and can realize data migration and synchronization between different storage devices; specifically, when the storage management module detects that the storage device has a device failure or the device is offline, it can change the mapping relationship between the unique identifier and the device adapter module, so that the target data corresponding to the unique identifier is stored in the storage device corresponding to the changed device adapter module (which can be an existing storage device or a temporarily added storage device), and is synchronously migrated back to the original storage device when the fault recovers, thereby providing the system with better fault resistance; when the storage management module detects that a new storage device is online, it can change the mapping relationship between the unique identifier and the device adapter module to migrate the data in the currently existing storage device to realize data management.

[0107] Therefore, the storage management module of this embodiment is based on a hot-swappable device adapter module, and can implement flexible operations on data between storage devices.

[0108] Second, based on the target service level to which the service request belongs, the mapping relationship between the unique identifier and the device adaptation module is changed.

[0109] The storage management module of this embodiment can also be customized according to business needs. For example, when the business need is to ensure the security of the data, or to specify the target business level in the business request, the storage management module can add a device adapter module corresponding to the unique identifier to provide redundant backup and other operations for business requests with higher importance; or when the business need is temporary storage, the device adapter module corresponding to the unique identifier can be changed to write the data to a temporary storage device to facilitate the sharing and use of temporary data.

[0110] Therefore, this embodiment can improve the flexibility of data management on the basis of managing massive unstructured data, and support customized data management according to different businesses;

[0111] Users can customize and expand storage resource deployment methods, data storage methods, and data management (such as version management, fragmented file integration, business information management, and large attachment transmission) based on business characteristics.

[0112] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data processing method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0113] This application also provides a data processing device, please refer to Figure 7 , the data processing device comprises:

[0114] The gateway module 10 is used to respond to the service request and determine the target service processing module based on the target service type to which the service request belongs, wherein each of the service processing modules corresponds to different data processing logic;

[0115] The business processing module 20 is used to process the target data corresponding to the business request through the target business processing module to obtain a processing result, wherein the target data is data obtained from a target storage device corresponding to the target business type, or data that needs to be stored in the target storage device.

[0116] The data processing device provided by the present application adopts the data processing method in the above embodiment, which can solve the technical problem of low business processing efficiency of the business system. Compared with the prior art, the beneficial effects of the data processing device provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and other technical features in the data processing device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0117] The present application provides a data processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data processing method in the above-mentioned embodiment 1.

[0118] The data processing device provided by the present application adopts the data processing method in the above embodiment to solve the technical problem of low business processing efficiency of the business system. Compared with the prior art, the beneficial effects of the data processing device provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and other technical features in the data processing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0119] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0120] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0121] The present application provides a computer-readable and writable storage medium having computer-readable and writable program instructions (ie, computer programs) stored thereon, wherein the computer-readable and writable program instructions are used to execute the data processing method in the above-mentioned embodiment.

[0122] The computer readable and writable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer readable and writable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer readable and writable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer readable and writable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0123] The computer-readable and writable storage medium may be included in the data processing device; or may exist independently without being installed in the data processing device.

[0124] The computer-readable and writable storage medium may carry one or more programs. When the one or more programs are executed by a data processing device, the data processing device executes the data processing method.

[0125] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0126] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to each embodiment of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0127] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0128] The readable and writable storage medium provided in the present application is a computer readable and writable storage medium, which stores computer readable and writable program instructions (i.e., computer programs) for executing the above-mentioned data processing method, and can solve the technical problem of low business processing efficiency of the business system. Compared with the prior art, the beneficial effects of the computer readable and writable storage medium provided in the present application are the same as the beneficial effects of the data processing method provided in the above-mentioned embodiment, and will not be described in detail here.

[0129] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned data processing method when executed by a processor.

[0130] The computer program product provided by this application can solve the technical problem of low business processing efficiency of the business system. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the data processing method provided by the above embodiment, which will not be repeated here.

[0131] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A data processing method, characterized in that: Applied to a data storage system, the data storage system includes multiple business processing modules and a storage management module, the storage management module is used to manage multiple storage devices, and the method includes: In response to the service request, determining a target service processing module based on the target service type to which the service request belongs, wherein each of the service processing modules corresponds to a different data processing logic; The target data corresponding to the business request is processed by the target business processing module to obtain a processing result, wherein the target data is data obtained from a target storage device corresponding to the target business type, or is data that needs to be stored in the target storage device.

2. The method according to claim 1, characterized in that The step of processing the target data corresponding to the business request by the target business processing module to obtain a processing result includes at least one of the following: Generate a unique identifier of the target data to be stored in the target storage device based on the target business type through the target business processing module, process the target data to obtain processed data, and store the processed data in the storage device corresponding to the unique identifier through the storage management module to obtain a processing result; The target business processing module obtains target data from the target storage device based on the unique identifier carried in the business request, processes the target data, and obtains a processing result.

3. The method according to claim 2, characterized in that The business request also carries a business requirement, wherein the business requirement includes storing target data, and the target data includes unstructured data; The step of processing the target data to obtain processed data comprises: Convert the unstructured data into a target data structure to obtain converted data; Based on the business requirements of storing the target data, generating target metadata corresponding to the unstructured data; The unique identifier, the converted data and the target metadata are packaged to obtain processed data.

4. The method according to claim 3, characterized in that The step of storing the processed data in a storage device corresponding to the unique identifier through the storage management module to obtain a processing result includes: Determining, by the storage management module, a target storage device based on a unique identifier in the processed data; The converted data and the target metadata are respectively stored in a storage device corresponding to the unique identifier.

5. The method according to claim 2, characterized in that The storage management module is provided with a plurality of device adapter modules, and the storage management module is respectively connected to each of the storage devices in communication via the plurality of device adapter modules; The step of storing the processed data in a storage device corresponding to the unique identifier through the storage management module to obtain a processing result includes: Determine, by the storage management module, a target device adapter module based on the unique identifier; The target device adapter module stores the processed data in the target storage device based on the preset device connection information of the target storage device to obtain a processing result.

6. The method according to any one of claims 1 to 5, characterized in that Before the step of processing the target data corresponding to the business request by the target business processing module to obtain a processing result, at least one of the following items is also included: When the storage management module detects that the storage device has changed, the mapping relationship between the unique identifier and the device adapter module is changed, and the change of the storage device includes at least one of a device failure and a device offline; Based on the target service level to which the service request belongs, the mapping relationship between the unique identifier and the device adaptation module is changed.

7. A data processing device, characterized in that: The data processing device comprises: A gateway module, configured to respond to a service request and determine a target service processing module based on a target service type to which the service request belongs, wherein each of the service processing modules corresponds to a different data processing logic; The business processing module is used to process the target data corresponding to the business request through the target business processing module to obtain a processing result, wherein the target data is data obtained from a target storage device corresponding to the target business type, or data that needs to be stored in the target storage device.

8. A data processing device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable and writable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 6 are implemented.

Images