A data processing method and device, electronic equipment and medium

Abstract

Embodiments of the present application disclose a data processing method and device, electronic equipment and medium. The method comprises: selecting a target scrambling strategy from candidate scrambling strategies in a scrambling strategy pool for target data; wherein the candidate scrambling strategy comprises a data writing mode for the target data; determining the number of storage areas required for data writing based on the target scrambling strategy, selecting a target storage area from available storage areas of a storage system based on the number of storage areas; writing the target data into the target storage area based on the target scrambling strategy, and recording metadata of the target data, an identifier of the target storage area and the target scrambling strategy in an index to query the written target data based on the index. The above scheme does not need to process the data itself with a key, can realize scrambling storage of the target data, reduces the calculation amount, and even if the storage system is open to the outside, the accurate target data cannot be sequentially read, improving the security of the target data.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, electronic device and medium. Background Technology

[0002] Data scrambling is a technique that transforms raw data into a seemingly random sequence through a predetermined algorithm. Its main purposes include ensuring data security and reliability, meeting technical requirements, and reducing error rates. Existing object storage data scrambling typically employs online or local encryption / decryption methods. During business layer read / write operations, a key must be obtained and data encryption / decryption calculations performed before business processing can proceed.

[0003] However, processing data according to the target's scrambling method requires additional encryption and decryption calculations for both reading and writing, affecting read and write performance. In addition, once the key is cracked, the data in the cluster will be completely leaked if stored one object at a time, threatening data security. Summary of the Invention

[0004] This application provides a data processing method, apparatus, electronic device, and medium that can perform scrambling processing on data without using a key.

[0005] According to one aspect of this application, a data processing method is provided, the method comprising:

[0006] For the target data, a target scrambling strategy is selected from the candidate scrambling strategies in the scrambling strategy pool; wherein, the candidate scrambling strategies include data writing methods;

[0007] Determine the number of storage areas required for data writing based on the target scrambling strategy, and select a target storage area from the available storage areas of the storage system based on the number of storage areas;

[0008] The target data is written into the target storage area based on the target scrambling strategy, and the metadata of the target data, the target storage area identifier, and the target scrambling strategy are recorded in the index so that the written target data can be queried based on the index.

[0009] According to one aspect of this application, a data processing apparatus is provided, the apparatus comprising:

[0010] The target scrambling strategy selection module is used to select a target scrambling strategy from the candidate scrambling strategies in the scrambling strategy pool for the target data; wherein, the candidate scrambling strategy includes a data writing method;

[0011] The target storage area determination module is used to determine the number of storage areas required for data writing based on the target scrambling strategy, and to select a target storage area from the available storage areas of the storage system based on the number of storage areas.

[0012] The writing module is used to write the target data into the target storage area based on the target scrambling strategy, and record the metadata of the target data, the target storage area identifier and the target scrambling strategy in the index, so as to query the written target data based on the index.

[0013] According to another aspect of this application, an electronic device is provided, the electronic device comprising:

[0014] At least one processor; and

[0015] A memory that is communicatively connected to at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the data processing method of any embodiment of the present application.

[0017] According to another aspect of this application, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement the data processing method of any embodiment of this application.

[0018] The technical solution of this application embodiment selects a target scrambling strategy from candidate scrambling strategies in a scrambling strategy pool for target data. The candidate scrambling strategies include data writing methods. The number of storage areas required for data writing based on the target scrambling strategy is determined, and a target storage area is selected from the available storage areas of the storage system based on the number of storage areas. The target data is written to the target storage area based on the target scrambling strategy, and the metadata of the target data, the target storage area identifier, and the target scrambling strategy are recorded in an index for querying the written target data based on the index. This solution eliminates the need for key processing of the data itself, enabling scrambling and storage of target data, reducing computational load, and improving the security of the target data by preventing sequential reading of accurate target data even if the storage system is publicly accessible.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0022] Figure 2 The target data provided in the embodiments of this application is written into the first schematic diagram;

[0023] Figure 3 This is a first schematic diagram illustrating the distribution of target data in the target storage area provided in the embodiments of this application;

[0024] Figure 4 This is a second schematic diagram illustrating the distribution of target data in the target storage area provided in the embodiments of this application;

[0025] Figure 5 The target data provided in the embodiments of this application is written into the second schematic diagram;

[0026] Figure 6 A schematic diagram of the selected target storage area provided in the embodiments of this application;

[0027] Figure 7 The target data provided in the embodiments of this application is written into the third schematic diagram;

[0028] Figure 8 A flowchart illustrating a data processing method provided in another embodiment of this application;

[0029] Figure 9 This is a schematic diagram illustrating the deletion of target data as provided in another embodiment of this application;

[0030] Figure 10 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0031] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0033] It should be noted that the terms "first," "second," "third," "fourth," "actual," "preset," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application. This embodiment is applicable to situations where data scrambling is required. The method can be executed by a data processing device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0035] S110. For the target data, select the target scrambling strategy from the candidate scrambling strategies in the scrambling strategy pool; wherein, the candidate scrambling strategy includes the data writing method.

[0036] In this application embodiment, the target data refers to the data that needs to be processed and, more specifically, the data that needs to be written into the storage system for storage. The scrambling strategy pool is a pre-built area that accommodates candidate scrambling strategies. Each candidate scrambling strategy is a pre-set scrambling strategy for the target data. Unlike conventional scrambling methods, candidate scrambling strategies include the data writing method for the target data, i.e., the method of storing the target data in the storage system. The data writing method refers to the restriction on the writing process of the target data, not the processing of the target data itself; therefore, it differs from conventional scrambling and encryption methods. Generally, at least two candidate scrambling strategies are used to allow for different selections at different writing cycles, increasing the disorder and complexity of the data writing method and increasing the difficulty of cracking.

[0037] In this embodiment, for the target data to be processed, a target scrambling strategy is selected from the candidate scrambling strategies in the scrambling strategy pool, and scrambling processing is performed based on the target scrambling strategy. The process of selecting the target scrambling strategy from the candidate scrambling strategies can be random selection, or the candidate scrambling strategies can be sorted and selected in a preset order. For example, for the first group of target data, the first candidate scrambling strategy is selected as the target scrambling strategy; for the second group of target data, the second candidate scrambling strategy is selected as the target scrambling strategy. Alternatively, for the first group of target data, the first candidate scrambling strategy is selected as the target scrambling strategy; for the second group of target data, the first candidate scrambling strategy is selected as the target scrambling strategy; for the third group of target data, the second candidate scrambling strategy is selected as the target scrambling strategy; and for the fourth group of target data, the second candidate scrambling strategy is selected as the target scrambling strategy. The method of sorting the candidate scrambling strategies and the preset order are not limited, as long as they are known.

[0038] S120. Determine the number of storage areas required for data writing based on the target scrambling strategy, and select a target storage area from the available storage areas of the storage system based on the number of storage areas.

[0039] In this context, a storage area can be a storage unit used for data storage within a storage system. In a distributed storage system, a storage area can be a node; in an object storage system, it can be a data object. An available storage area is a storage area that, while not currently storing valid target data, still supports writing target data. The target storage area is an available storage area selected from the available storage areas for storing the target data. The number of storage areas is determined according to the theoretical requirements of the data writing method in the target scrambling strategy. For example, in a backup-based data writing method, at least two storage areas are required; in a distributed parity-checking data writing method, at least three storage areas are required.

[0040] In this embodiment, after selecting a target scrambling strategy, the number of storage areas required for data writing based on the target scrambling strategy is determined. Specifically, for each candidate scrambling strategy, the number of storage areas required for data writing based on the candidate scrambling strategy can be predetermined, and the correspondence between candidate scrambling strategies and the number of storage areas can be recorded. After selecting a target scrambling strategy, the number of storage areas corresponding to the target scrambling strategy can be determined based on the recorded correspondence.

[0041] After determining the number of storage areas, a target storage area can be selected from the storage system. Specifically, based on the number of storage areas, a target storage area is selected from the available storage areas in the storage system. An available storage area with a number greater than or equal to the number of storage areas can be selected as the target storage area. In this embodiment, the location of the selected target storage area is not limited; it can be any location in the storage system, as long as the identifier of the target storage area is recorded to determine which target storage areas the target data resides in.

[0042] S130. Write the target data into the target storage area based on the target scrambling strategy, and record the metadata of the target data, the target storage area identifier and the target scrambling strategy in the index, so as to query the written target data based on the index.

[0043] Metadata of the target data refers to descriptive data about the target data, such as its name, identifier, size, encoding format, and data type. The target storage area identifier is used to distinguish different target storage areas. An index is a structure or tool used for quickly locating and accessing data; in this embodiment, it can be a secondary index, which is another type of index in the database besides the primary key index, used to accelerate queries of non-primary key fields.

[0044] In this embodiment, target data can be written to the target storage area based on a target scrambling strategy. Since the target scrambling strategy is only one candidate scrambling strategy selected for this specific writing of target data, it is impossible to know which storage areas the target data is stored in without knowing the target scrambling strategy. Therefore, the effect of scrambling and protecting the target data is achieved. In addition, by using a different writing method as a candidate scrambling strategy, scrambling of the target data can be achieved without encryption or other complex processing, reducing the computational load.

[0045] In this embodiment, after writing target data into the target storage area, the metadata of the target data, the target storage area identifier, and the target scrambling strategy need to be recorded in an index. This allows for subsequent queries of the target data to identify which storage areas the target data is stored in by querying the index, and to determine the appropriate query method based on the target scrambling strategy. This eliminates the need to decrypt the target data to query and process the scrambled target data. Specifically, as shown... Figure 2As shown, assuming the selected target storage areas are data0, data2, and data3, the target data is written into data0, data2, and data3, and the metadata of the written target data, the target storage area identifier, and the target scrambling policy are recorded in the index accordingly. For example, the record is "taget data1{"link_data": "0,2,3"}{"policy": "RAID0,32"}", where target data1 is the metadata of the target data, the target storage area identifier is 0,2,3, the target scrambling policy is RAID0, and the stripe width is 32K.

[0046] The process in this embodiment is for each write cycle. Each write cycle can be determined as follows: if the amount of target data is less than or equal to the capacity of the target storage area, then the writing process of the target data is considered as one write cycle; if the amount of target data is greater than the capacity of the target storage area, then the process of filling the target storage area with target data is considered as one write cycle. The remaining unwritten target data is then written by the execution process of subsequent write cycles.

[0047] The technical solution of this application embodiment selects a target scrambling strategy from candidate scrambling strategies in a scrambling strategy pool for target data. The candidate scrambling strategies include data writing methods. The number of storage areas required for data writing based on the target scrambling strategy is determined, and a target storage area is selected from the available storage areas of the storage system based on the number of storage areas. The target data is written to the target storage area based on the target scrambling strategy, and the metadata of the target data, the target storage area identifier, and the target scrambling strategy are recorded in an index for querying the written target data based on the index. This solution eliminates the need for key processing of the data itself, enabling scrambling and storage of target data, reducing computational load, and improving the security of the target data by preventing sequential reading of accurate target data even if the storage system is publicly accessible.

[0048] As a non-limiting implementation, the data writing method includes independent disk redundant array storage and stripe depth; writing the target data into the target storage area based on the target scrambling strategy includes:

[0049] Target data is written to the target storage area based on the independent disk redundant array storage method in the target scrambling strategy; the amount of target data written to a single target storage area each time satisfies the stripe depth in the target scrambling strategy.

[0050] Redundant Array of Independent Disks (RAID) is a data storage technology that combines multiple physical disks into a single logical unit to provide higher performance, reliability, or capacity than a single disk. RAID technology achieves data distribution and redundant storage across multiple disks through mechanisms such as data striping, mirroring, and parity checking. RAID technologies include RAID 0, RAID 1, RAID 5, and RAID 10. Stripe depth refers to the size of the data block occupied by each stripe on a single disk when using striping technology. In this embodiment, stripe depth refers to the amount of data written to a single target storage area in a single operation. In a predetermined candidate scrambling strategy, different candidate scrambling strategies include different writing methods, which are different RAID storage methods and different stripe depths. The target scrambling strategy includes a specific RAID storage method and a specific stripe depth. Data can be written to the target storage area based on the RAID storage method in the target scrambling strategy, and the amount of target data written to a single target storage area each time satisfies the stripe depth in the target scrambling strategy. For example, such as Figure 3 As shown, assuming the selected target storage areas are target storage area 0, target storage area 2, and target storage area 3, the independent disk redundant array storage mode is RAID0, the stripe depth is 4K, and the target data is divided into 16 4K data blocks, then data block 1 is written to target storage area 0, data block 2 is written to target storage area 2, data block 3 is written to target storage area 3, data block 4 is written to target storage area 0, data block 5 is written to target storage area 2, data block 6 is written to target storage area 3, and so on, until all 15 data blocks are written to the target storage area. Figure 4As shown, assuming a stripe width of 8K, data blocks 1 and 2 are written to target storage area 0, data blocks 3 and 4 are written to target storage area 2, data blocks 5 and 6 are written to target storage area 3, data blocks 7 and 8 are written to target storage area 0, data blocks 9 and 10 are written to target storage area 2, data blocks 11 and 12 are written to target storage area 3, and so on, until all 15 data blocks are written to the target storage area. The beneficial effect of the above scheme is that, in the case of RAID storage, it can form a variety of candidate scrambling strategies based on rich RAID storage methods. Furthermore, by coordinating different stripe depth settings, it can further enrich the scrambling forms, forming different combinations of RAID storage methods and stripe depths as candidate scrambling strategies. This ensures that scrambling is achieved even when target data is written in a specific data writing method, reducing the computational load required for encrypting and decrypting the target data itself, and improving the security of the target data. Even if the storage area is exposed, target data written in different writing methods cannot be read sequentially.

[0051] As a non-limiting implementation, determining the number of storage areas required for data writing based on the target scrambling strategy includes:

[0052] The minimum number of storage areas theoretically required for data writing is determined based on the data writing method in the target scrambling strategy.

[0053] The number of storage areas required for data writing according to the target scrambling strategy is determined to be greater than or equal to the minimum number of storage areas.

[0054] In this scheme, the data writing method in the target scrambling strategy is known, and the theoretically required number of storage areas is also determined by this method. Depending on the data writing method itself, the minimum number of storage areas theoretically required for data writing can be determined based on the data writing method in the target scrambling strategy. For example, RAID 0 requires a minimum of 2 storage areas, RAID 1 requires 2, RAID 5 requires 3, and RAID 10 requires 4. The number of storage areas required for data writing according to the target scrambling strategy is determined to be greater than or equal to the minimum number of storage areas to meet the requirements of data writing based on the target scrambling strategy's data writing method. The beneficial effect of this scheme is that it can adaptively select the required number of storage areas based on the data writing method in the target scrambling strategy, satisfying the need to write target data based on the target scrambling strategy's data writing method, thus providing the basic conditions for implementing scrambling based on the target scrambling strategy.

[0055] As a non-limiting implementation, selecting a target storage area from the available storage areas of the storage system based on the number of storage areas includes:

[0056] Randomly select a target storage area that meets the required number of storage areas from the available storage areas of the storage system, or select a target storage area that meets the required number of storage areas from the available storage areas of the storage system, wherein the target storage area contains at least two available storage areas whose storage addresses are not adjacent;

[0057] Accordingly, after writing the target data into the target storage area based on the target scrambling strategy, the method further includes:

[0058] The target storage area is marked as an unusable storage area, and the storage areas not marked as unusable storage areas are treated as usable storage areas.

[0059] Specifically, in the process of selecting the target storage area, a target storage area that meets the required number of storage areas can be randomly selected from the available storage areas of the storage system. This creates interference for unauthorized access to target data by malfunctioning devices, avoiding sequential storage and facilitating the reading of target data by malfunctioning devices. The "randomness" in the above scheme can refer to the random selection of the location of each target storage area, or it can refer to the random selection of the locations of different groups of target storage areas for different target data. For example, such as... Figure 5 As shown, for the first target data, three target storage areas (data0, data1, and data2) can be selected using any of the methods described above to store the first target data. For the second target data, four target storage areas (data3, data4, data5, and data6) can be selected to store the second target data. For the third target storage area, three target storage areas (data11, data12, data13, data14, and data15) can be selected to store the third target data. And so on. After writing the target data into the target storage area, the target storage area needs to be marked as unavailable, and the storage areas not marked as unavailable should be designated as available storage areas so that subsequent selection of target storage areas can be made from the available storage areas. Alternatively, target storage areas can be arbitrarily selected without specifying a location as the target storage area for each target data.

[0060] In another feasible approach, a target storage area is selected from the available storage areas of the storage system to meet the required number of storage areas. This target storage area must contain at least two available storage areas with non-adjacent storage addresses. This avoids the risk of contiguous, ordered storage making it easier for abnormal devices to read the target data, increasing the scrambling force and improving the security of the target data. For example, such as... Figure 6As shown, for the first target data, three target storage areas (data0, data2, and data3) can be selected using any of the methods described above to store the first target data. For the second target data, three target storage areas (data1, data7, and data10) can be selected to store the second target data. Similarly, for the third target data, three target storage areas (data4, data6, and data9) are selected to store the third target data. For the fourth target data, three target storage areas (data5, data8, and data11) are selected to store the fourth target data, and so on. After writing the target data into the target storage area, it is necessary to mark the target storage area as an unavailable storage area and treat the storage areas not marked as unavailable storage areas as available storage areas so that subsequent selection of target storage areas can be made from the available storage areas. Figure 6 As shown, the white storage area is the available storage area, and the storage areas with other colors are unavailable storage areas, which already contain the target data. Figure 6 The annotation method shown is merely a way to facilitate understanding; in reality, storage areas can be annotated in the form of text, tables, etc. For example, as shown in Table 1, available and unavailable storage areas are distinguished so that the target storage area can be selected from the available storage areas later.

[0061] Table 1

[0062]

[0063] Based on the examples above, after selecting the target storage area to store the target data, the metadata corresponding to the target data, the target storage area identifier, and the target scrambling strategy can be added. For example... Figure 7 As shown, for the first target data, after writing it to the target storage area, the metadata of the first target data, the target storage area identifier, and the target scrambling policy are recorded in the index to form index1, for example, "target data 1{"link_data": "0,2,3"}{"policy": "RAID0,32"}". For the second target data, after writing it to the target storage area, the metadata of the second target data, the target storage area identifier, and the target scrambling policy are recorded in the index to form index2, for example, "target data2{"link_data": "1,7,10"}{"policy": "RAID5,16"}". And so on.

[0064] Figure 8This is a flowchart illustrating a data processing method according to another embodiment of this application. This embodiment is an optimization based on the above embodiment; solutions not described in detail in this embodiment are found in the above embodiment. Figure 8 As shown, the method in this embodiment of the application specifically includes the following steps:

[0065] S210. For the target data, select a target scrambling strategy from the candidate scrambling strategies in the scrambling strategy pool; wherein, the candidate scrambling strategy includes a data writing method.

[0066] S220. Determine the number of storage areas required for data writing based on the target scrambling strategy, and select a target storage area from the available storage areas of the storage system based on the number of storage areas.

[0067] S230. Write the target data into the target storage area based on the target scrambling strategy, and record the metadata of the target data, the target storage area identifier, and the target scrambling strategy in the index.

[0068] S240. Query the target storage area identifier and the target scrambling strategy corresponding to the metadata in the index.

[0069] For example, after writing target data to the target storage area, subsequent applications may need to query the target data. During the query process, the metadata of the target data to be queried is determined. The target storage area identifier and target scrambling policy corresponding to the metadata are queried from the index, so that the target data can be accurately queried based on the target storage area identifier and the target scrambling policy. For example, assuming the metadata of the target data is target data 2, the index in the example above can be used to query the target storage area identifier "link_data": "1,7,10" and the target scrambling policy "policy": "RAID5,16" corresponding to target data 2. That is, the target data is stored in target storage area 1, target storage area 7, and target storage area 10. The target scrambling policy is RAID5 storage with a stripe depth of 16K.

[0070] S250. Query the target storage area according to the target storage area identifier, and query the target data corresponding to the metadata in the target storage area based on the target scrambling strategy.

[0071] For example, the target storage area can be queried in the storage area of ​​the storage system based on the target storage area identifier to determine the specific storage area where the target data is located, and the target data corresponding to the metadata can be queried in the target storage area based on the target scrambling strategy.

[0072] In this embodiment, the data writing method includes independent disk redundant array storage and stripe depth; querying the target data corresponding to the metadata in the target storage area based on the target scrambling strategy includes:

[0073] Based on the independent disk redundant array storage method in the target scrambling strategy, the target storage area is traversed to query the target data corresponding to the metadata;

[0074] During the traversal, stripe data that meets the stripe depth is read from a single target storage area, and the stripe data read in sequence is merged to obtain the target data.

[0075] For example, assuming a query from the index for metadata target data 1 retrieves the target storage area "link_data": "0,2,3" and the target scrambling policy "policy": "RAID0,4", then the target storage areas can be identified as target storage area 0, target storage area 2, and target storage area 3. The target data is then searched within these three target storage areas. The target scrambling policy is based on RAID0 write mode with a stripe depth of 4K. When searching for the target data in the three target storage areas, the RAID0 approach means reading data sequentially from the three target storage areas, reading 4K blocks of data at a time. Finally, all read data blocks are concatenated to obtain the target data. Figure 3 As shown, the reading process according to the above scheme involves reading 4K data block 1 from target storage area 0, 4K data block 2 from target storage area 2, 4K data block 3 from target storage area 3, 4K data block 4 from target storage area 0, and so on, until data block 15 is read. Then, the data is concatenated in sequence to obtain the target data. Figure 4 As shown, assuming the target scrambling strategy is based on RAID0 write mode, the stripe depth is 8K, and each data block is 4K, the reading according to the above scheme is to read 8K data blocks 1 and 2 in target storage area 0, 8K data blocks 3 and 4 in target storage area 2, 8K data blocks 5 and 6 in target storage area 3, 8K data blocks 7 and 8 in target storage area 0, and so on, until data block 15 is read, and then they are concatenated in order to obtain the target data.

[0076] In this embodiment of the application, the process of deleting the target data from the storage system includes:

[0077] Delete the metadata, the target storage area identifier, and the record corresponding to the target scrambling policy from the index, and remove the unavailable storage area marked for the target storage area and / or mark the target storage area as an available storage area.

[0078] For example, in the process of deleting target data from the storage system, in order to reduce processing volume and difficulty, the target data in the target storage area of ​​the storage system can be removed directly from the index instead of being deleted. This involves deleting the metadata, target storage area identifier, and records corresponding to the target scrambling strategy, and removing the unavailable storage area markers for the target storage area and / or marking the target storage area as available, thus releasing the target storage area on the record level. In this case, when writing target data again subsequently, the target data can be directly written to the released target storage area, overwriting the previously written target data, achieving the effect of deleting the previously written target data. Figure 9 As shown, in Figure 7 Deleting index4 from the index actually releases storage areas 5, 8, and 11, forming... Figure 9 The effects available in the three storage areas mentioned above.

[0079] This application provides a data processing method. In the process of querying target data in a storage system, the target storage area identifier and target scrambling strategy can be queried in the index based on the metadata, thereby achieving accurate reading of the target data. The target data can be read smoothly without decrypting the target data or performing other scrambling reverse operations, which reduces the processing volume and reading difficulty, and does not compromise the security of the target data.

[0080] Figure 10 This is a schematic diagram of a data processing apparatus provided in an embodiment of this application. The apparatus can execute the data processing method provided in any embodiment of this application, and possesses the corresponding functional modules and beneficial effects for executing the method. For example... Figure 10 As shown, the device includes:

[0081] The target scrambling strategy selection module 310 is used to select a target scrambling strategy from the candidate scrambling strategies in the scrambling strategy pool for the target data; wherein, the candidate scrambling strategy includes a data writing method;

[0082] The target storage area determination module 320 is used to determine the number of storage areas required for data writing based on the target scrambling strategy, and to select a target storage area from the available storage areas of the storage system based on the number of storage areas.

[0083] The writing module 330 is used to write the target data into the target storage area based on the target scrambling strategy, and record the metadata of the target data, the target storage area identifier and the target scrambling strategy in an index, so as to query the written target data based on the index.

[0084] In this embodiment, the data writing method includes independent disk redundant array storage and stripe depth; the writing module 330 writes the target data into the target storage area based on the target scrambling strategy, including:

[0085] Target data is written to the target storage area based on the independent disk redundant array storage method in the target scrambling strategy; the amount of target data written to a single target storage area each time satisfies the stripe depth in the target scrambling strategy.

[0086] In this embodiment of the application, the target storage area determination module 320 determines the number of storage areas required for data writing based on the target scrambling strategy, including:

[0087] The minimum number of storage areas theoretically required for data writing is determined based on the data writing method in the target scrambling strategy.

[0088] The number of storage areas required for data writing according to the target scrambling strategy is determined to be greater than or equal to the minimum number of storage areas.

[0089] In this embodiment of the application, the target storage area determination module 320 selects a target storage area from the available storage areas of the storage system based on the number of storage areas, including:

[0090] Randomly select a target storage area that meets the required number of storage areas from the available storage areas of the storage system, or select a target storage area that meets the required number of storage areas from the available storage areas of the storage system, wherein the target storage area contains at least two available storage areas whose storage addresses are not adjacent;

[0091] Accordingly, after writing the target data into the target storage area based on the target scrambling strategy, the method further includes:

[0092] The target storage area is marked as an unusable storage area, and the storage areas not marked as unusable storage areas are treated as usable storage areas.

[0093] In this embodiment of the application, the device further includes:

[0094] The indexing module is used to query the target storage area identifier and the target scrambling strategy corresponding to the metadata in the index;

[0095] The query module is used to query the target storage area based on the target storage area identifier, and to query the target data corresponding to the metadata in the target storage area based on the target scrambling strategy.

[0096] In this embodiment, the data writing method includes independent disk redundant array storage and stripe depth; the query module queries the target data corresponding to the metadata in the target storage area based on the target scrambling strategy, including:

[0097] Based on the independent disk redundant array storage method in the target scrambling strategy, the target storage area is traversed to query the target data corresponding to the metadata;

[0098] During the traversal, stripe data that meets the stripe depth is read from a single target storage area, and the stripe data read in sequence is merged to obtain the target data.

[0099] In this embodiment of the application, the device further includes:

[0100] The index deletion module is used to delete the metadata, the target storage area identifier, and the record corresponding to the target scrambling strategy from the index, and to remove the unavailable storage area marked for the target storage area and / or mark the target storage area as an available storage area.

[0101] The data processing apparatus provided in this application embodiment can execute a data processing method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of executing the method.

[0102] Figure 11 A schematic diagram of an electronic device 10, which can be used to implement embodiments of this application, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0103] like Figure 11As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0104] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of monitors, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless data processing transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0105] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as data processing methods.

[0106] In some embodiments, the data processing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the data processing method by any other suitable means (e.g., by means of firmware).

[0107] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0108] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0109] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0110] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0111] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0112] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0113] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired information of the technical solution of this application can be achieved, and this is not limited herein.

[0114] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The method includes: For the target data, a target scrambling strategy is selected from the candidate scrambling strategies in the scrambling strategy pool; wherein, the candidate scrambling strategies include data writing methods; Determine the number of storage areas required for data writing based on the target scrambling strategy, and select a target storage area from the available storage areas of the storage system based on the number of storage areas; The target data is written into the target storage area based on the target scrambling strategy, and the metadata of the target data, the target storage area identifier, and the target scrambling strategy are recorded in the index so that the written target data can be queried based on the index.

2. The method according to claim 1, characterized in that, The data writing method includes independent disk redundant array storage and stripe depth; writing the target data to the target storage area based on the target scrambling strategy includes: Target data is written to the target storage area based on the independent disk redundant array storage method in the target scrambling strategy; the amount of target data written to a single target storage area each time satisfies the stripe depth in the target scrambling strategy.

3. The method according to claim 1, characterized in that, Determining the number of storage areas required for data writing based on the target scrambling strategy includes: The minimum number of storage areas theoretically required for data writing is determined based on the data writing method in the target scrambling strategy. The number of storage areas required for data writing according to the target scrambling strategy is determined to be greater than or equal to the minimum number of storage areas.

4. The method according to claim 1, characterized in that, Selecting a target storage area from the available storage areas of the storage system based on the number of storage areas includes: Randomly select a target storage area that meets the required number of storage areas from the available storage areas of the storage system, or select a target storage area that meets the required number of storage areas from the available storage areas of the storage system, wherein the target storage area contains at least two available storage areas whose storage addresses are not adjacent; Accordingly, after writing the target data into the target storage area based on the target scrambling strategy, the method further includes: The target storage area is marked as an unusable storage area, and the storage areas not marked as unusable storage areas are treated as usable storage areas.

5. The method according to claim 1, characterized in that, The method further includes: Query the target storage area identifier and the target scrambling strategy corresponding to the metadata in the index; The target storage area is queried based on the target storage area identifier, and the target data corresponding to the metadata is queried in the target storage area based on the target scrambling strategy.

6. The method according to claim 5, characterized in that, The data writing method includes independent disk redundant array storage and stripe depth; querying the target data corresponding to the metadata in the target storage area based on the target scrambling strategy includes: Based on the independent disk redundant array storage method in the target scrambling strategy, the target storage area is traversed to query the target data corresponding to the metadata; During the traversal, stripe data that meets the stripe depth is read from a single target storage area, and the stripe data read in sequence is merged to obtain the target data.

7. The method according to claim 5, characterized in that, The process of deleting the target data from the storage system includes: Delete the metadata, the target storage area identifier, and the record corresponding to the target scrambling policy from the index, and remove the unavailable storage area marked for the target storage area and / or mark the target storage area as an available storage area.

8. A data processing apparatus, characterized in that, The device includes: The target scrambling strategy selection module is used to select a target scrambling strategy from the candidate scrambling strategies in the scrambling strategy pool for the target data; wherein, the candidate scrambling strategy includes a data writing method; The target storage area determination module is used to determine the number of storage areas required for data writing based on the target scrambling strategy, and to select a target storage area from the available storage areas of the storage system based on the number of storage areas. The writing module is used to write the target data into the target storage area based on the target scrambling strategy, and record the metadata of the target data, the target storage area identifier and the target scrambling strategy in the index, so as to query the written target data based on the index.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data processing method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the data processing method according to any one of claims 1-7.

Images