Data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a data processing method and device, electronic equipment and a storage medium, and relates to the technical field of computers, in particular to the technical fields of distributed storage systems, data management and the like. According to the specific implementation scheme, a to-be-processed data object is received; if the to-be-processed data object meets the preset packaging rule, packaging the to-be-processed data object to obtain packaged data, and storing the packaged data to the proxy server; recording forward index information according to the packaged data, and returning the forward index information to the controller so as to respond to a data retrieval request of the user through the controller; and recording reverse index information according to the packaged data, and storing the reverse index information into a target scanning database so as to manage the data through the target scanning database. The data processing efficiency and the utilization rate of the storage space are effectively improved, and the convenience of data management and the maintainability of the system are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to the field of distributed storage systems, data management and data processing technology, and especially to data processing methods, devices, electronic devices and storage media. Background Art

[0002] In today's big data era, distributed storage services have become a key technology for processing and storing massive amounts of data. Object storage organizes and manages data objects in a flat architecture, showing significant advantages in large-scale data storage. This architecture supports horizontal expansion of the system and can easily cope with the rapid growth of data volume while maintaining efficient data access speed.

[0003] However, the massive small file scenario poses a challenge to object storage services. In this scenario, data exists in the form of a large number of small files. The average size of each file is small, but the overall number is large, resulting in frequent write and delete requests. This high IO (Input / Output) pressure not only increases the complexity of metadata management, but may also reduce system performance and data access efficiency. Therefore, there is an urgent need for a new technical solution to optimize the storage and management of massive small files to improve the performance and scalability of distributed storage services when processing such data. Summary of the invention

[0004] The present disclosure provides a data processing method, device, electronic device and storage medium.

[0005] According to one aspect of the present disclosure, a data processing method is provided, the method comprising:

[0006] Receive the data object to be processed;

[0007] If the data object to be processed meets the preset packaging rule, the data object to be processed is packaged to obtain the packaged data and store it in the proxy server;

[0008] Recording forward index information according to the packaged data, and returning the forward index information to the controller, so as to respond to the user's data retrieval request through the controller;

[0009] The reverse index information is recorded according to the packaged data, and the reverse index information is stored in a target scanning database, so as to manage the data through the target scanning database.

[0010] According to another aspect of the present disclosure, there is provided a data processing device, comprising:

[0011] A receiving module, used for receiving a data object to be processed;

[0012] A packaging module, used for packaging the data object to be processed if it meets the preset packaging rules, obtaining the packaged data and storing it in the proxy server;

[0013] A first recording module, used for recording forward index information according to the packaged data, and returning the forward index information to the controller, so as to respond to the user's data retrieval request through the controller;

[0014] The second recording module is used to record the reverse index information according to the packaged data, and store the reverse index information in the target scanning database, so as to manage the data through the target scanning database.

[0015] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any method in any of the above technical solutions.

[0019] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods described in the above technical solutions.

[0020] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements any one of the methods described in the above technical solutions when executed by a processor.

[0021] The present disclosure provides a data processing method, device, equipment and storage medium. The present disclosure effectively improves the efficiency of data processing and the utilization rate of storage space by receiving data objects to be processed and intelligently packaging them when preset packaging rules are met. At the same time, forward index information is generated for the packaged data to quickly respond to user retrieval requests, and reverse index information is created to optimize data management. In this way, not only the speed and accuracy of data retrieval are enhanced, but also the convenience of data management and the maintainability of the system are improved, thereby providing users with an efficient and reliable data storage and access solution.

[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0024] Figure 1 is a system block diagram corresponding to the data processing method in the embodiment of the present disclosure;

[0025] Figure 2 is a schematic diagram of the steps of the data processing method in an embodiment of the present disclosure;

[0026] Figure 3 is a schematic diagram of a process of packaging data in an embodiment of the present disclosure;

[0027] Figure 4 is a schematic diagram of the process of updating reverse index information in an embodiment of the present disclosure;

[0028] Figure 5 is a flowchart of a data processing method in an embodiment of the present disclosure;

[0029] Figure 6 A principle block diagram of a data processing device in an embodiment of the present disclosure;

[0030] Figure 7 It is a block diagram of an electronic device used to implement the data processing method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] To understand the present disclosure as a whole, see Figure 1 As shown, Figure 1It is a system block diagram corresponding to the data processing method in the embodiment of the present disclosure. The system includes a controller, a packaging module, a proxy server, and a metadata database. The corresponding working principle is as follows: the client (c1, c2, c3...cn) sends a data write request to the controller, and the controller distributes the request to different storage buckets (bucket1, bucket2, bucket3...) according to the content of the request and the system rules. Then, the corresponding packaging module processes these requests, performs data packaging and index recording. Finally, the packaged data is written to the storage system through the proxy, and the metadata storage is updated to maintain the index and access path of the data, thereby achieving efficient data management and fast access.

[0033] The present disclosure provides a data processing method, see Figure 2 As shown, Figure 2 is a schematic diagram of the steps of the data processing method in the embodiment of the present disclosure, which can be applied to Figure 1 The packaging module in the method includes:

[0034] Step S201: receiving a data object to be processed.

[0035] Specifically, "receiving data objects to be processed" refers to obtaining data objects that are about to be further processed, and these data may come from the controller. These data objects may include but are not limited to files uploaded by users, database records, log data, temporary files, backup data, and user-generated content. By receiving data objects to be processed, the server can evaluate these data objects according to preset packaging rules, such as file size, type or source, to determine whether they need to be packaged into larger data units to optimize storage efficiency and reduce IO (Input / Output) pressure on the storage system. This process is the basis for achieving efficient data management and rapid access, and provides the necessary input for subsequent data processing, packaging, index creation and storage operations.

[0036] In this way, by receiving the data objects to be processed, the data entities to be packaged, indexed and stored can be efficiently introduced, laying the foundation for subsequent data optimization processing, thereby improving the data management efficiency and retrieval speed of the entire storage system, and ensuring the continuity of the data processing process and the agility of the system response.

[0037] Step S202: If the data object to be processed meets the preset packaging rule, the data object to be processed is packaged to obtain the packaged data and store it in the proxy server.

[0038] Specifically, "preset packaging rules" refer to a series of predefined standards or conditions to determine which data objects should be grouped together for packaging processing. When these data objects meet preset rules such as file size, quantity or specific time intervals, they will be automatically combined into a larger data unit, namely "packaged data". This process involves the collection, evaluation and merging of data, aiming to optimize storage efficiency and reduce the complexity of system processing by reducing the number of individual data objects, thereby improving the performance of the entire storage and retrieval process.

[0039] In this way, by packaging the data objects to be processed that meet the preset packaging rules, the number of single small files in the storage system can be effectively reduced, the demand for storage resources and the frequency of IO operations can be reduced, thereby improving storage efficiency and system performance, while simplifying data management and facilitating subsequent data retrieval and maintenance.

[0040] Step S203, recording forward index information according to the packaged data, and returning the forward index information to the controller, so as to respond to the user's data retrieval request through the controller.

[0041] Specifically, "forward index information" refers to a data structure that records the specific location and related information of each original data object in the packaged data unit, which includes the data object's identifier, offset in the packaged file, and other key attributes. The implementation process of this solution includes: after the data is packaged, these forward index information are automatically generated and sent back to the controller. The controller uses this information to respond to the user's data retrieval request, quickly locate and extract specific data objects in the packaged file, thereby improving the efficiency and accuracy of data retrieval. This process optimizes the data access process, making data retrieval in large-scale data environments faster and more reliable.

[0042] In this way, by recording forward index information according to the packaged data and returning the forward index information to the controller, the controller can respond to the user's data retrieval request, thus achieving efficient management and fast access to the packaged data. This process not only optimizes the performance of data retrieval and reduces the search time, but also improves the flexibility and accuracy of data processing, so that even after the data is packaged and stored, the required specific data objects can be quickly located and extracted, thereby improving the response speed of the entire system and user experience.

[0043] Step S204, recording reverse index information according to the packaged data, and storing the reverse index information in a target scanning database, so as to manage the data through the target scanning database.

[0044] Specifically, "reverse index information" refers to a data structure that records detailed information about each original data object in the packaged data unit and its location in the packaged file. This index allows the system to quickly trace back to a single data object from the packaged file. "Target scanning database" refers to a database that specifically stores this reverse index information to support data management operations.

[0045] The implementation process of this solution includes: after the data is packaged, the reverse index entries containing the metadata and location information of each data object are automatically generated, and these entries are stored in the target scanning database. In this way, when data recovery, maintenance or analysis is required, these index information can be used to efficiently access and manage the data in the packaged file, ensuring the consistency and traceability of the data, thereby optimizing the data management process and improving the overall performance of the system.

[0046] In this way, by recording the reverse index information and storing it in the target scanning database, it is possible to ensure that even after the data is packaged and stored, individual data objects can be quickly located and accessed through the reverse index, thereby improving the efficiency of data retrieval, enhancing the flexibility and accuracy of data management, and making data maintenance, recovery and analysis more efficient and reliable.

[0047] The present disclosure provides a data processing method, device, equipment and storage medium. The present disclosure effectively improves the efficiency of data processing and the utilization rate of storage space by receiving data objects to be processed and intelligently packaging them when preset packaging rules are met. At the same time, forward index information is generated for the packaged data to quickly respond to user retrieval requests, and reverse index information is created to optimize data management. In this way, not only the speed and accuracy of data retrieval are enhanced, but also the convenience of data management and the maintainability of the system are improved, thereby providing users with an efficient and reliable data storage and access solution.

[0048] In some optional embodiments, the packaging rule includes any of the following conditions:

[0049] If the arrival time of the data object to be processed exceeds the preset maximum packing interval from the current time, the packing rule is met;

[0050] If the number of packaged items of the data object to be processed exceeds the preset package threshold, the package rule is met;

[0051] If the package size of the data object to be processed exceeds the preset size threshold, the package rule is met.

[0052] Specifically, the packaging rules of this solution are dynamic packaging rules set based on three dimensions: time, quantity, and size. Among them, the "preset maximum packaging interval" refers to a set time threshold. Data objects that have not been processed beyond this time will be deemed to meet the packaging conditions. The "preset packaging threshold" and "preset size threshold" are the preset upper limits of the number and size, respectively. When the total number or total size of the data objects to be processed exceeds these thresholds, it is also deemed to meet the packaging conditions. The implementation process of this solution involves monitoring the arrival time, quantity, and size of the data objects to be processed. Once any condition is triggered, these data objects will be automatically combined into larger data units for packaging, so as to optimize storage efficiency and reduce the system's overhead in processing small files. This intelligent packaging mechanism helps to improve the flexibility of data processing and the overall performance of the system.

[0053] In this way, by setting dynamic packaging rules based on time intervals, quantity and size, it is possible to intelligently identify and process the data objects to be packaged, ensuring that the data is packaged in a timely manner, thereby optimizing the use of storage resources, reducing the system's overhead for small file operations, and improving data processing efficiency. This method not only reduces fragmentation in the storage system, but also enhances the system's scalability and response speed, making data management more efficient and orderly.

[0054] In some optional embodiments, after receiving the data object to be processed, the method further includes:

[0055] After receiving the data writing request sent by the user, encapsulate the writing request, the data object to be processed and the expected response information into the context to obtain the encapsulated context information;

[0056] The encapsulated context information is added to the cache queue, and is allocated to different processing queues according to the hash key calculation results corresponding to the encapsulated context information;

[0057] Determine whether to package the data objects to be processed in different queues according to the preset packaging rules.

[0058] Specifically, "data write request sent by user" refers to the request submitted by the user to the system to store data. After receiving these requests, the system will encapsulate the request details, the related "data objects to be processed" (that is, the specific data content to be stored) and the expected "response information" (the information that the system will return to the user after processing the request) into a "context". This context is a data structure containing all necessary information for subsequent processing. Subsequently, this encapsulated context information is added to the "cache queue", which is a queue for temporary storage of requests to ensure that the requests can be processed in order. Next, the system will assign the requests to different "processing queues" based on the calculation results of the "hash key" related to the context information. The hash key is a value derived from the context information through a hash function and is used to determine which processing queue the request should be processed by. Finally, the system determines whether the data objects to be processed in each processing queue meet the packaging conditions based on the "preset packaging rules". These rules may be based on factors such as the size, quantity or time of the data objects. Data objects that meet the conditions will be packaged together to optimize storage efficiency and reduce system overhead. The entire process is designed to increase the speed and efficiency of data processing while maintaining flexibility and reliability in data management.

[0059] To facilitate understanding of the contents of the embodiments of the present disclosure, see Figure 3 As shown, Figure 3 It is a flow chart of data packaging in the embodiment of the present disclosure. After receiving the data write request (data request) sent by the user, the write request, the data object to be processed and the expected response information are cached (buffer), and then the data are exchanged to the local (local) storage. Then, the data are divided and encapsulated into multiple context (context) information, each context information is associated with a key (key), and is distributed to different processing queues (queues) according to the hash key calculation result corresponding to the encapsulated context information. Check whether the packaging conditions are met according to the preset packaging rules. If they are met, generate a snapshot of the current queue and submit it to thread pool 1; thread pool 1 divides the context in the queue into one or more packaging units, so that each unit meets the packaging threshold, and submits the divided large package unit to thread pool 2 to perform packaging (do pack) operation. This process embodies an efficient data processing and packaging mechanism based on multi-threading and conditional triggering.

[0060] In this way, after receiving the user's data write request, the request details, the data object to be processed and the expected response information are encapsulated into context information, and the information is intelligently assigned to the corresponding processing queue according to the hash key, and then the data is packaged according to the preset packaging rules, thereby optimizing the data processing process. This method not only improves the efficiency and accuracy of data processing, but also enhances the scalability and response speed of the system by reducing redundant operations and optimizing resource allocation, providing users with faster and more reliable data storage services.

[0061] In some optional embodiments, recording forward index information according to the packaged data and returning the forward index information to the controller so as to respond to the user's data retrieval request through the controller includes:

[0062] If the data object to be processed is included in the package file, the detailed information of the data object to be processed in the package file is recorded to obtain forward index information, and the detailed information includes a file identifier and an offset;

[0063] The forward index information is returned to the controller so that the controller can respond to the user's data retrieval request.

[0064] Specifically, when the data objects to be processed are determined to be included in the package file, the detailed information of these data objects in the package file is recorded to generate the so-called "forward index information". These detailed information include the unique "file identifier" of each data object and their "offset" in the package file, that is, the starting position of the data object in the file. This process ensures that even if the data is packaged and stored, the system can accurately track and locate each individual data object. Subsequently, the system sends this "forward index information" back to the "controller", which uses this information to respond to the user's data retrieval request, quickly locate and extract the data objects required by the user, thereby improving the speed and accuracy of data retrieval and optimizing the entire data retrieval process.

[0065] In this way, by recording the detailed information of the data object to be processed after it is included in the packaged file, including the file identifier and offset, generating forward index information, and returning this information to the controller, this solution significantly improves the efficiency and accuracy of data retrieval. This mechanism enables quick location and access to specific files even when the data is packaged, thereby optimizing the data management and access process, and improving the system's response speed and user satisfaction.

[0066] In some optional embodiments, recording reverse index information according to the packaged data and storing the reverse index information in a target scanning database so as to manage the data through the target scanning database includes:

[0067] Record the metadata of the data object to be processed and the location information of the data object to be processed in the packaged data;

[0068] Creating a reverse index entry according to the metadata and the location information to obtain reverse index information;

[0069] The reverse index information is stored in the target scanning database so as to manage the data through the target scanning database.

[0070] Specifically, the "metadata" of these data objects is first recorded, that is, information describing data attributes, such as file name, size and type, as well as their specific "location information" in the "packaged data", such as the offset in the packaged file. Using these metadata and location information, the system then creates "reverse index entries", which is a data structure that allows the system to trace back to a single data object from the packaged file. Finally, these "reverse index information" are stored in the "target scan database", which is a database dedicated to storing index information, which makes data management more efficient because the system can quickly access and manage data objects in the packaged file through indexes, thereby optimizing the data retrieval and maintenance process.

[0071] In this way, by recording the metadata of the data object to be processed and its location information in the packaged data, this solution can create accurate reverse index entries and then generate reverse index information. Storing this reverse index information in the target scanning database greatly enhances the efficiency and accuracy of data management, enables the system to quickly retrieve and access a single data object in the packaged file, optimizes the process of data recovery, maintenance and analysis, and improves the maintainability and reliability of the entire storage system.

[0072] In some optional embodiments, the method further comprises:

[0073] All data objects to be processed are scanned, and the collected scan information table is recorded in the target scan database, wherein the scan information table includes metadata of the data objects to be processed and location information of the objects to be processed in the package file.

[0074] Specifically, all data objects to be processed are scanned and a scanning operation is performed, that is, a comprehensive check is performed on these data objects to collect their metadata (such as file name, size, type, etc.) and specific location information in the packaged file (such as offset). This information is summarized into a "scan information table", which records the key attributes and storage location of each data object in detail. Subsequently, the system "records this scan information table into the target scan database", which is specially designed to store and manage this scan information so that data objects can be quickly retrieved and accessed when needed. This process not only improves the efficiency of data management, but also provides important reference information for data maintenance, backup and recovery.

[0075] In this way, by fully scanning all data objects to be processed and recording the scan information table containing metadata and location information in the target scan database, this solution effectively enhances the traceability and manageability of data. This approach not only ensures that the detailed information of the data object is properly preserved, but also optimizes the data retrieval process and improves the efficiency of data recovery and analysis, thus providing a solid foundation for data integrity and system reliability.

[0076] In some optional embodiments, the method further comprises:

[0077] If a data deletion request is received, the metadata of the data object to be deleted is deleted, and the reverse index information is updated.

[0078] Specifically, when a data deletion request is received, that is, a user or application initiates an operation to remove a specific data object from the storage system, the system will perform two key steps: first, "delete the metadata of the data object to be deleted", which means that the system will remove the descriptive information related to the data object from the metadata database, such as the file name, size, and creation date; second, "update the reverse index information", which involves modifying the reverse index stored in the target scan database to ensure that the index information in the system reflects this change, that is, remove or mark the deleted data object from the index, so as to maintain the consistency and accuracy of the data. In addition, it should be noted that when deleting, the metadata of the data object is deleted first, and then the bitmap information of the reverse index is updated, and the data is not deleted directly. The data is not deleted until all the metadata recorded in the bitmap has been deleted. This process ensures that the data deletion operation is not only processed at the metadata level, but also updated accordingly in the index structure of the entire system, maintaining the synchronization of the system status and the effectiveness of data management.

[0079] To facilitate understanding of the solutions of the embodiments of the present disclosure, the following examples are given. Figure 4 , Figure 4It is a flowchart of updating reverse index information in an embodiment of the present disclosure. Assume that there is a cloud storage service, and a user uploads multiple files (such as object1 to object4). Each file is assigned a unique identifier (slice_key) (such as slice_key1 to slice_key4) and recorded in the reverse index information, wherein the reverse index information also includes the upload time and the source list. When object3 is deleted or overwritten, the system will mark the slice_key corresponding to object3 as 0 in the reverse index information, indicating that the file no longer exists. That is, if the requested file has been deleted or overwritten, the reverse index will provide corresponding status information, and the system can notify the user that the file does not exist or has been updated. If the source list is too long, the system will only save two records in order to optimize storage. In this way, whenever the user requests to access these files, the system can quickly locate the status and location of the file by querying the reverse index information, and even if the file is deleted or overwritten, it can accurately reflect the latest data status.

[0080] In this way, by timely deleting the metadata of the data object to be deleted and updating the reverse index information when receiving a data deletion request, the consistency of data in the storage system and the accuracy of the index are ensured, thereby improving the efficiency and reliability of data management, enabling the system to accurately reflect the latest status of the data and support effective data recovery and audit operations. This process optimizes the life cycle management of data and reduces potential errors and security risks caused by data inconsistency.

[0081] In order to understand the technical solution of this application as a whole, see Figure 5 , Figure 5 It is a flow chart of the data processing method in the embodiment of the present disclosure. Specifically, first, the user submits a data object upload (put object) request through the controller to upload the data object to be processed; the controller forwards the request to the packaging module (packagemodule), the packaging module processes the data and generates packaged data, and then writes the data to the storage system through the proxy server (proxy); then, the packaging module writes a scan information table (scan table) in the target scan database (tafdb) to record the scan information, and writes a reverse index table in the tafdb database to track the packaged data objects; finally, the packaging module returns the forward index information to the controller, and the controller writes this information into the metadata database (meta), thereby completing the entire data packaging and index update process, and returning the operation result to the user.

[0082] The following describes an embodiment of the device of the present application, which can be used to execute the data processing method in the above embodiment of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the data processing method in the above embodiment of the present application.

[0083] The present disclosure also provides a data processing device 600, such as Figure 6 As shown, including:

[0084] Receiving module 601, used for receiving a data object to be processed;

[0085] The packaging module 602 is used to package the data object to be processed if it meets the preset packaging rules, obtain the packaged data and store it in the proxy server;

[0086] A first recording module 603, used to record forward index information according to the packaged data, and return the forward index information to the controller, so as to respond to the user's data retrieval request through the controller;

[0087] The second recording module 604 is used to record the reverse index information according to the packaged data, and store the reverse index information in the target scanning database, so as to manage the data through the target scanning database.

[0088] In some optional embodiments, the packaging rule set in the packaging module 602 includes any of the following conditions:

[0089] If the arrival time of the data object to be processed exceeds the preset maximum packing interval from the current time, the packing rule is met;

[0090] If the number of packaged items of the data object to be processed exceeds the preset package threshold, the package rule is met;

[0091] If the package size of the data object to be processed exceeds the preset size threshold, the package rule is met.

[0092] In some optional embodiments, after receiving the data object to be processed, the packaging module 602 is further configured to:

[0093] After receiving the data writing request sent by the user, encapsulate the writing request, the data object to be processed and the expected response information into the context to obtain the encapsulated context information;

[0094] The encapsulated context information is added to the cache queue, and is allocated to different processing queues according to the hash key calculation results corresponding to the encapsulated context information;

[0095] Determine whether to package the data objects to be processed in different queues according to the preset packaging rules.

[0096] In some optional embodiments, the first recording module 603 records the forward index information according to the packaged data, and returns the forward index information to the controller, so as to respond to the user's data retrieval request through the controller, including:

[0097] If the data object to be processed is included in the package file, the detailed information of the data object to be processed in the package file is recorded to obtain forward index information, and the detailed information includes a file identifier and an offset;

[0098] The forward index information is returned to the controller so that the controller can respond to the user's data retrieval request.

[0099] In some optional embodiments, the second recording module 604 records the reverse index information according to the packaged data, and stores the reverse index information in the target scanning database, so as to manage the data through the target scanning database, including:

[0100] Record the metadata of the data object to be processed and the location information of the data object to be processed in the packaged data;

[0101] Creating a reverse index entry according to the metadata and the location information to obtain reverse index information;

[0102] The reverse index information is stored in the target scanning database so as to manage the data through the target scanning database.

[0103] In some optional embodiments, the device also includes a scanning module for scanning all data objects to be processed and recording the collected scanning information table into a target scanning database, wherein the scanning information table includes metadata of the data objects to be processed and location information of the objects to be processed in the packaged file.

[0104] In some optional embodiments, the device further includes an updating module for deleting metadata of the data object to be deleted and updating the reverse index information if a data deletion request is received.

[0105] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0106] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0107] Figure 4A schematic block diagram of an example electronic device 400 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0108] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0109] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0110] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as data processing methods. For example, in some embodiments, the data processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the applet distribution described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the data processing method in any other appropriate manner (e.g., by means of firmware).

[0111] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0112] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0113] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0114] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0115] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0116] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0117] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0118] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data processing method, the method comprising: Receive the data object to be processed; If the data object to be processed meets the preset packaging rule, the data object to be processed is packaged to obtain the packaged data and store it in the proxy server; Recording forward index information according to the packaged data, and returning the forward index information to the controller, so as to respond to the user's data retrieval request through the controller; Reverse index information is recorded according to the packaged data, and the reverse index information is stored in a target scanning database, so as to manage the data through the target scanning database.

2. The method according to claim 1, wherein: The packaging rule includes any of the following conditions: If the arrival time of the data object to be processed exceeds the preset maximum packing interval from the current time, the packing rule is met; If the number of packaged items of the data object to be processed exceeds a preset package threshold, the package rule is satisfied; If the package size of the data object to be processed exceeds a preset size threshold, the package rule is satisfied.

3. The method according to claim 1, wherein: After receiving the data object to be processed, the method further includes: After receiving a data write request sent by a user, encapsulate the write request, the data object to be processed, and the expected response information into a context to obtain encapsulated context information; Adding the encapsulated context information to a cache queue, and distributing it to different processing queues according to the hash key calculation results corresponding to the encapsulated context information; It is determined whether to package the to-be-processed data objects in different queues according to the preset packaging rule.

4. The method according to claim 1, wherein: The step of recording forward index information according to the packaged data and returning the forward index information to the controller so as to respond to the user's data retrieval request through the controller includes: If the data object to be processed is included in the package file, the detailed information of the data object to be processed in the package file is recorded to obtain forward index information, wherein the detailed information includes a file identifier and an offset; The forward index information is returned to the controller so as to respond to the user's data retrieval request through the controller.

5. The method according to claim 1, wherein: The step of recording reverse index information according to the packaged data and storing the reverse index information in a target scanning database so as to manage the data through the target scanning database includes: Recording metadata of the data object to be processed and location information of the data object to be processed in the packaged data; Creating a reverse index entry according to the metadata and the location information to obtain reverse index information; The reverse index information is stored in the target scanning database so as to manage the data through the target scanning database.

6. The method according to any one of claims 1 to 5, wherein: The method further comprises: All the data objects to be processed are scanned, and the collected scan information table is recorded in the target scan database, wherein the scan information table includes the metadata of the data objects to be processed and the location information of the objects to be processed in the package file.

7. The method according to any one of claims 1 to 5, wherein: The method further comprises: If a data deletion request is received, the metadata of the data object to be deleted is deleted, and the reverse index information is updated.

8. A data processing device, comprising: A receiving module, used for receiving a data object to be processed; A packaging module, used for packaging the data object to be processed if it meets the preset packaging rules, obtaining the packaged data and storing it in the proxy server; A first recording module, used for recording forward index information according to the packaged data, and returning the forward index information to the controller, so as to respond to the user's data retrieval request through the controller; The second recording module is used to record the reverse index information according to the packaged data, and store the reverse index information in a target scanning database, so as to manage the data through the target scanning database.

9. The device according to claim 8, wherein: The packaging rules set in the packaging module include any of the following conditions: If the arrival time of the data object to be processed exceeds the preset maximum packing interval from the current time, the packing rule is met; If the number of packaged items of the data object to be processed exceeds a preset package threshold, the package rule is satisfied; If the package size of the data object to be processed exceeds a preset size threshold, the package rule is satisfied.

10. The device according to claim 8, wherein: After receiving the data object to be processed, the packaging module is further used to: After receiving a data write request sent by a user, encapsulate the write request, the data object to be processed, and the expected response information into a context to obtain encapsulated context information; Adding the encapsulated context information to a cache queue, and distributing it to different processing queues according to the hash key calculation results corresponding to the encapsulated context information; It is determined whether to package the to-be-processed data objects in different queues according to the preset packaging rule.

11. The device according to claim 8, wherein: The first recording module records forward index information according to the packaged data, and returns the forward index information to the controller so as to respond to the user's data retrieval request through the controller, including: If the data object to be processed is included in the package file, the detailed information of the data object to be processed in the package file is recorded to obtain forward index information, wherein the detailed information includes a file identifier and an offset; The forward index information is returned to the controller so as to respond to the user's data retrieval request through the controller.

12. The device according to claim 8, wherein: The second recording module records the reverse index information according to the packaged data, and stores the reverse index information in a target scanning database, so as to manage the data through the target scanning database, including: Recording metadata of the data object to be processed and location information of the data object to be processed in the packaged data; Creating a reverse index entry according to the metadata and the location information to obtain reverse index information; The reverse index information is stored in the target scanning database so as to manage the data through the target scanning database.

13. The device according to any one of claims 8 to 12, wherein: The device also includes a scanning module for scanning all data objects to be processed and recording the collected scanning information table into the target scanning database, wherein the scanning information table includes metadata of the data objects to be processed and location information of the objects to be processed in the packaged file.

14. The device according to any one of claims 8 to 12, wherein: The device also includes an updating module, which is used to delete the metadata of the data object to be deleted and update the reverse index information if a data deletion request is received.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

17. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images