An information processing method, apparatus, device, and computer-readable storage medium

Abstract

Embodiments of the present application disclose an information processing method, which comprises: obtaining to-be-processed data of a plurality of objects; performing verification on the to-be-processed data of the plurality of objects in parallel through a target instruction to obtain a verification value corresponding to each to-be-processed data, and storing the to-be-processed data of each object; wherein the target instruction is an instruction for performing multi-data operation; for the to-be-processed data of each object, in a case where the to-be-processed data has been stored and the to-be-processed data is determined to be legal based on the verification value corresponding to the to-be-processed data, storing metadata of the to-be-processed data. Embodiments of the present application also disclose an information processing device, equipment and computer readable storage medium.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of object storage, and particularly relates to an information processing method and device, equipment and a computer readable storage medium. BACKGROUND

[0002] Object storage is object / file-oriented, mass Internet storage, which can also be directly referred to as cloud storage. Object storage is to manage data as an object. When object storage is performed, user data uploaded by a user is usually read first, then the user data is checked, the user data is stored, and finally metadata of the user data is stored. However, the object storage manner can only process data of one object each time, and it takes a long time and is low in efficiency when object storage is performed on mass data. SUMMARY

[0003] To solve the above technical problems, the embodiments of the present application provide an information processing method, device, equipment and computer readable storage medium, which solve the problem of long time and low efficiency when object storage is performed on mass data in the related art.

[0004] The technical scheme of the present application is implemented as follows:

[0005] An information processing method, characterized in that the method comprises:

[0006] obtaining to-be-processed data of a plurality of objects;

[0007] checking the to-be-processed data of the plurality of objects in parallel through a target instruction to obtain a check value corresponding to each to-be-processed data, and storing the to-be-processed data of each object; wherein the target instruction is an instruction for performing multi-data operation;

[0008] For the to-be-processed data of each object, when the to-be-processed data has been stored and it is determined that the to-be-processed data is legal based on the check value corresponding to the to-be-processed data, storing metadata of the to-be-processed data.

[0009] An information processing device, characterized in that the device comprises:

[0010] an obtaining unit, configured to obtain to-be-processed data of a plurality of objects;

[0011] a processing unit, configured to check the to-be-processed data of the plurality of objects in parallel through a target instruction to obtain a check value corresponding to each to-be-processed data, and store the to-be-processed data of each object; wherein the target instruction is an instruction for performing multi-data operation;

[0012] The processing unit is further configured to, for each object, store metadata of the to-be-processed data of the object in a case that the to-be-processed data of the object has been stored and that the to-be-processed data of the object is determined to be legal based on the check value corresponding to the to-be-processed data of the object.

[0013] An information processing device, comprising: a processor, a memory and a communication bus;

[0014] The communication bus is configured to realize the communication connection between the processor and the memory.

[0015] The processor is configured to execute an information processing program stored in the memory to realize the steps of the information processing method.

[0016] A computer readable storage medium storing one or more programs, which can be executed by one or more processors to realize the steps of the information processing method.

[0017] The information processing method, device, equipment and computer readable storage medium provided by the embodiments of the present application acquire to-be-processed data of a plurality of objects, perform parallel verification on the to-be-processed data of the plurality of objects through a target instruction to obtain a check value corresponding to each to-be-processed data, and store the to-be-processed data of each object, wherein the target instruction is an instruction for performing multi-data operation; for each object, in a case that the to-be-processed data of the object has been stored and that the to-be-processed data of the object is determined to be legal based on the check value corresponding to the to-be-processed data of the object, metadata of the to-be-processed data of the object is stored. In this way, in the face of massive data, the to-be-processed data of a plurality of objects can be verified in parallel through an instruction for performing multi-data operation, the simultaneous batch verification of a plurality of to-be-processed data is realized, the time spent in object storage of massive data is reduced, the efficiency of object storage is improved, and the problem of long time consumption and low efficiency in object storage of massive data in related technologies is solved. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 A flowchart of an information processing method provided by an embodiment of the present application;

[0019] Figure 2 A flowchart of another information processing method provided by an embodiment of the present application;

[0020] Figure 3 A flowchart of still another information processing method provided by an embodiment of the present application;

[0021] Figure 4 A flowchart of a verification process in an information processing method provided by an embodiment of the present application;

[0022] Figure 5 A flowchart of object storage in the related art provided for the embodiments of the present application is shown in FIG. 1.

[0023] Figure 6 A working diagram of an information processing system provided for the embodiments of the present application is shown in FIG. 2.

[0024] Figure 7 A structural diagram of an information processing apparatus provided for the embodiments of the present application is shown in FIG. 3.

[0025] Figure 8 A structural diagram of an information processing device provided for the embodiments of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application.

[0027] It should be understood that the specific embodiments described herein are merely intended to explain the present application, and are not intended to limit the present application.

[0028] The embodiments of the present application provide an information processing method, which can be applied to an information processing device. Referring to FIG. 1, the method comprises the following steps: Figure 1

[0029] Step 101: Obtain the to-be-processed data of a plurality of objects.

[0030] In the embodiments of the present application, the information processing device can be a server in an object storage system; the to-be-processed data can be user data carried in a storage request received by the information processing device; wherein the object in the plurality of objects refers to a basic unit of object storage. Each object is composed of an object key, an object value and metadata; the object key is a unique identifier of the object in a storage bucket; the object value refers to user data; the metadata refers to metadata of the user data, which can be data describing the attributes of the user data. The to-be-processed data refers to the user data (i.e. the object value) of the object.

[0031] Specifically, the information processing device can receive the to-be-processed data of a plurality of objects sent by a user terminal. In a feasible implementation manner, the to-be-processed data can be at least one of a document, a picture and a video uploaded by a user. The metadata can be information such as the last modification time and size of the user data, and the initial check value of the to-be-processed data can be carried in the metadata, or the initial check value of the to-be-processed data can not be carried.

[0032] Step 102: Perform check on the to-be-processed data of the plurality of objects in parallel through a target instruction, obtain a check value corresponding to each to-be-processed data, and store the to-be-processed data of each object. ​

[0033] The target instruction is an instruction for performing multi-data operation.

[0034] In the embodiment of the present application, the target instruction can be invoked, and the integrity of the to-be-processed data of each object is checked in parallel by executing the target instruction, to obtain a check value corresponding to each to-be-processed data, and the to-be-processed data of each object is stored in the database. The checking and storing of the to-be-processed data are parallel processing. The multi-data operation on the to-be-processed data of multiple objects is performed by the target instruction, so as to check the integrity of the to-be-processed data of multiple objects.

[0035] In step 103, for the to-be-processed data of each object, if the to-be-processed data has been stored and the to-be-processed data is determined to be legal based on the check value corresponding to the to-be-processed data, the metadata of the to-be-processed data is stored.

[0036] The metadata of the to-be-processed data refers to the metadata of the user data, and the metadata is carried in the storage request.

[0037] In the embodiment of the present application, for the to-be-processed data of each object, it can be determined whether the initial check value corresponding to the to-be-processed data is carried in the metadata of the to-be-processed data. If the initial check value is carried in the metadata, it can be determined whether the initial check value is consistent with the check value obtained by checking the to-be-processed data. If the initial check value is consistent with the check value, it is determined that the to-be-processed data has not been tampered with, that is, the to-be-processed data is legal, and then the metadata can be stored. After that, information for notifying that the data is stored successfully can be sent to the user terminal. The data includes the to-be-processed data and the metadata. Of course, if the initial check value is not consistent with the check value obtained by checking the to-be-processed data, the metadata of the to-be-processed data is not stored, but information for notifying that the data is stored unsuccessfully is sent to the user terminal. If the to-be-processed data has been stored, the check value corresponding to the to-be-processed data is obtained by checking, and the initial check value corresponding to the to-be-processed data is not carried in the metadata, it is determined that the to-be-processed data is legal by default, and the metadata and the check value obtained by checking the to-be-processed data are stored.

[0038] It should be noted that the location of storing the metadata is different from the location of storing the to-be-processed data. In a feasible implementation manner, the information processing device can store the to-be-processed data in a storage pool and store the metadata of the to-be-processed data in a database.

[0039] The information processing method provided by the embodiment of the present application comprises the following steps: obtaining to-be-processed data of a plurality of objects; performing verification on the to-be-processed data of the plurality of objects in parallel through a target instruction to obtain a verification value corresponding to each to-be-processed data, and storing the to-be-processed data of each object; wherein the target instruction is an instruction for performing multi-data operation; for the to-be-processed data of each object, if the to-be-processed data has been stored and it is determined that the to-be-processed data is legal based on the verification value corresponding to the to-be-processed data, storing the metadata of the to-be-processed data. In this way, in the face of massive data, the to-be-processed data of the plurality of objects can be verified in parallel through the instruction for performing multi-data operation, the plurality of to-be-processed data is verified at the same time in batches, the time spent in object storage of the massive data is reduced, the efficiency of object storage is improved, and the problem of long time consumption and low efficiency in object storage of massive data in the related art is solved.

[0040] Based on the foregoing embodiment, the embodiment of the present application provides an information processing method, as shown in the accompanying drawings, Figure 2 The method comprises the following steps:

[0041] Step 201, the information processing device receives a plurality of processing requests.

[0042] The plurality of processing requests are used to request storage of to-be-processed data of a plurality of objects.

[0043] In the embodiment of the present application, the processing request can be a storage request for storing to-be-processed data sent by a user through a terminal; wherein the users corresponding to the plurality of processing requests can be the same user or different users. The processing request carries to-be-processed data and metadata of the to-be-processed data; wherein the to-be-processed data can be user data.

[0044] Step 202, the information processing device obtains to-be-processed data of a plurality of objects from the plurality of processing requests.

[0045] In the embodiment of the present application, each processing request can be parsed to obtain to-be-processed data of an object from each processing request. The object is for object storage; the object comprises to-be-processed data and metadata of the to-be-processed data.

[0046] In a feasible implementation manner, the metadata comprises size, name, initial verification value of the to-be-processed data, addressing mode of to-be-processed sub-data, etc.

[0047] Step 203, the information processing device continuously segments the to-be-processed data based on a target size threshold for each to-be-processed data of an object, in the case that to-be-processed sub-data is obtained in each segmentation, performs verification on each to-be-processed sub-data of the object through a target instruction to obtain a verification value, and stores the to-be-processed sub-data.

[0048] In the embodiment of the present application, the target size threshold can be pre-set; in a feasible implementation manner, the target size threshold can be 4K. For the to-be-processed data of each object, the to-be-processed data can be first segmented based on the target size threshold to obtain a first to-be-processed sub-data, and then the to-be-processed data remaining after the first segmentation can be second segmented to obtain a second to-be-processed sub-data, until the to-be-processed sub-data remaining after the Mth segmentation is smaller than or equal to the target size threshold. In this way, the to-be-processed sub-data obtained is a data segment smaller than the object particle.

[0049] In a feasible implementation manner, the size of each to-be-processed sub-data can be 4K.

[0050] It should be noted that the to-be-processed data is continuously segmented based on the target size threshold, that is, the next segmentation is performed immediately after the to-be-processed sub-data is obtained by the current segmentation, without the need to perform the next segmentation after the to-be-processed sub-data is processed subsequently after the current segmentation, thereby reducing the time for segmenting the to-be-processed data.

[0051] In the embodiment of the present application, the target instruction can be an instruction capable of performing multi-data operation obtained by calling a target function; the to-be-processed sub-data of each object in the plurality of objects can be verified in parallel through the target instruction to obtain a verification value corresponding to each to-be-processed sub-data. Wherein, for the to-be-processed data of an object, the verification value corresponding to the to-be-processed data is obtained after the plurality of to-be-processed sub-data corresponding to the to-be-processed data of the object are verified; for a to-be-processed sub-data, the to-be-processed sub-data is also stored when the to-be-processed sub-data is verified, so that the verification and storage of the to-be-processed data can be processed in parallel.

[0052] It should be noted that step 203 can be implemented by steps A1-A2.

[0053] Step A1, the information processing device segments the to-be-processed data of each object based on the target size threshold to obtain an Nth to-be-processed sub-data.

[0054] Wherein, N is a positive integer.

[0055] In this embodiment, when N is 1, the data to be processed can be segmented for the first time based on the target size threshold to obtain the first sub-data to be processed. When N is greater than 1, the data to be processed is segmented for the Nth time based on the target size threshold. This can be understood as segmenting the remaining data to be processed after the (N-1)th segmentation based on the target size threshold to obtain the Nth sub-data to be processed. The size of the sub-data to be processed is less than or equal to the target size threshold.

[0056] Step A2: The information processing device stores the Nth sub-data to be processed and verifies the Nth sub-data to be processed through the target instruction. At the same time, based on the target size threshold, the data to be processed after the Nth segmentation is segmented for the N+1th time until all data to be processed is verified and stored.

[0057] Where N is a positive integer; the data to be processed after the Nth segmentation is the remaining data to be processed after the Nth segmentation.

[0058] In this embodiment, while storing the Nth sub-data to be processed, it is also necessary to verify the Nth sub-data to be processed by the target instruction, and to perform the N+1th segmentation of the data to be processed after the Nth segmentation based on the target size threshold; that is, the verification and storage of the Nth sub-data to be processed, and the N+1th segmentation of the data to be processed after the Nth segmentation are performed simultaneously, realizing the asynchronous concurrent processing of different sub-data to be processed of the same object, reducing the time for processing the data to be processed of the same object.

[0059] It should be noted that the verification of the Nth sub-data to be processed by the target instruction in step A2 can be achieved through steps a1 to a3:

[0060] Step a1: The information processing device generates a verification request for the sub-data to be processed for each object.

[0061] The verification request contains sub-data to be processed.

[0062] In this embodiment, for each object's unprocessed sub-data, a verification request can be generated to verify the unprocessed sub-data; the number of unprocessed sub-data for each object is the same as the number of verification requests. Each object's unprocessed data can be divided into multiple unprocessed sub-data, and each unprocessed sub-data corresponds to one verification request; therefore, it can be understood that each unprocessed data can correspond to multiple verification requests, or each object can correspond to multiple verification requests.

[0063] The information processing device determines, based on the position of the to-be-processed sub-data in the corresponding to-be-processed data, the Nth check request corresponding to the Nth to-be-processed sub-data of the plurality of objects from the plurality of check requests stored in each request queue.

[0064] In the embodiments of the present application, for each to-be-processed data, based on the position of the to-be-processed sub-data in the to-be-processed data, the Nth check request corresponding to the Nth to-be-processed sub-data is determined from the sorted check requests. In this way, the Nth check request corresponding to multiple objects can be obtained at one time.

[0065] In a feasible implementation, for the to-be-processed data of the ith object, the check request corresponding to the to-be-processed sub-data of the ith object in the request queue can be sorted based on the position of the to-be-processed sub-data of the ith object in the to-be-processed data of the ith object, and the check request corresponding to the to-be-processed sub-data with the most advanced position is determined from the sorted check requests to obtain the Nth check request corresponding to the ith object.

[0066] Step a3, the information processing device determines, based on the plurality of Nth check requests of the plurality of objects in each request queue, the Nth check value corresponding to each Nth to-be-processed sub-data by simultaneously checking the Nth to-be-processed sub-data carried in the Nth check request of the plurality of objects in each request queue through the target instruction.

[0067] Wherein, N is a positive integer greater than 1.

[0068] In the embodiments of the present application, in response to the Nth check request corresponding to the plurality of objects, the Nth check value corresponding to each Nth to-be-processed sub-data is obtained by simultaneously checking the Nth to-be-processed sub-data in the Nth check request of each request queue in the plurality of request queues through the target instruction. In this way, simultaneous mass checking of the to-be-processed data of the plurality of objects is realized, and the efficiency of checking the to-be-processed data of the plurality of objects is improved. For a request queue, the Nth check request corresponding to the plurality of objects in the request queue can be processed to obtain the check value corresponding to the Nth to-be-processed sub-data carried in each Nth check request. When checking the Nth check request in the plurality of request queues simultaneously, mass checking of the to-be-processed data of the plurality of objects can be realized.

[0069] Step 204, the information processing device stores the metadata of the to-be-processed data for each object's to-be-processed data, and in the case that the to-be-processed data has been stored and the to-be-processed data is determined to be legal based on the corresponding check value.

[0070] The information processing method provided in the embodiments of the present application can verify the to-be-processed data of multiple objects in parallel through the execution of the instruction for multiple data operations, thereby realizing the batch verification of multiple to-be-processed data at the same time, reducing the time spent on the object storage of massive data, improving the efficiency of object storage, and solving the problem of long time consumption and low efficiency in the related art when facing massive data and performing object storage.

[0071] Based on the foregoing embodiments, the embodiments of the present application provide an information processing method, as shown in Figure 3 The method comprises the following steps:

[0072] Step 301, the information processing device receives multiple processing requests.

[0073] The multiple processing requests are used to request to store the to-be-processed data of multiple objects.

[0074] Step 302, the information processing device obtains the to-be-processed data of multiple objects from the multiple processing requests.

[0075] Step 303, for the to-be-processed data of each object, the information processing device performs the Nth segmentation on the to-be-processed data based on a target size threshold, to obtain an Nth to-be-processed sub-data.

[0076] Step 304, the information processing device stores the Nth to-be-processed sub-data.

[0077] Step 305, the information processing device generates a verification request for the to-be-processed sub-data of each object.

[0078] The verification request carries the to-be-processed sub-data.

[0079] Step 306, the information processing device determines the Nth verification request corresponding to the Nth to-be-processed sub-data of multiple objects from the multiple verification requests stored in each request queue based on the position of the to-be-processed sub-data in the corresponding to-be-processed data.

[0080] Step 307, the information processing device verifies the Nth to-be-processed sub-data carried in the multiple Nth verification requests of multiple objects in each request queue through a target instruction at the same time based on the multiple Nth verification requests of multiple objects in each request queue, to obtain an Nth verification value corresponding to each Nth to-be-processed sub-data.

[0081] In the embodiments of the present application, for each request queue, the Nth verification request of each object in the multiple objects can be determined from the request queue, and the Nth to-be-processed sub-data carried in the multiple Nth verification requests in the multiple request queues can be verified through a target instruction at the same time, to obtain an Nth verification value corresponding to each Nth to-be-processed sub-data.

[0082] It should be noted that the step 307 of checking the Nth to-be-processed sub-data carried in the Nth check request of the plurality of objects in each request queue by the target instruction simultaneously can be implemented by steps B1-B2.

[0083] The step B1 comprises: invoking a vector extension instruction set by the information processing device through a target function.

[0084] In the embodiment of the present application, the vector extension instruction set can be invoked by the target function. The target function is a target library function, and the vector extension instruction set can be an Advanced Vector Extensions (AVX).

[0085] In a feasible implementation manner, the target library function uses functions such as “md5_ctx_mgr_init()”, “md5_ctx_mgr_submit()” and “md5_ctx_mgr_flush()” in the isa-l_crypto library.

[0086] The step B2 comprises: performing integrity check on the Nth to-be-processed sub-data carried in the Nth check request of the plurality of objects in each request queue by the information processing device through the instruction in the vector extension instruction set, to obtain the Nth check value corresponding to each Nth to-be-processed sub-data.

[0087] In the embodiment of the present application, the thread corresponding to each request queue in the plurality of request queues can simultaneously invoke the instruction in the vector extension instruction set to process the Nth check request of the plurality of objects in the plurality of request queues, so as to simultaneously process the Nth check request of the plurality of objects in the plurality of request queues. Each request queue has the Nth check request of the plurality of objects.

[0088] Specifically, for each request queue, in response to the Nth check request of the plurality of objects in the request queue, the instruction in the vector extension instruction set is used to simultaneously perform integrity check on the Nth to-be-processed sub-data carried in the plurality of Nth check requests by using a digest algorithm, to obtain the Nth check value corresponding to each Nth to-be-processed sub-data.

[0089] In a feasible implementation manner, the digest algorithm can be a Message Digest Algorithm MD5 (MD5).

[0090] It should be noted that for each piece of data to be processed, when N=1, an information digest algorithm can be used to process the first piece of data to be processed to obtain the check value corresponding to the first piece of data to be processed; when N is a positive integer greater than 1, an information digest algorithm can be used to process the check value corresponding to the (N-1)th piece of data to be processed and the Nth piece of data to be processed to obtain the check value corresponding to the Nth piece of data to be processed; that is, when checking the data to be processed, specifically, multiple pieces of data to be processed obtained from the segmentation of the data to be processed are checked. When all multiple pieces of data to be processed obtained from the segmentation of the data to be processed have been checked, it is determined that the data to be processed has been checked, and the check value of the data to be processed can be obtained; where the check value of the data to be processed is the check value corresponding to the last piece of data to be processed.

[0091] In one feasible implementation, when the data to be processed is divided into five parts, six sub-data to be processed can be obtained. The check value corresponding to the last sub-data to be processed, i.e. the sixth sub-data to be processed, can be used as the check value of the data to be processed.

[0092] Step 308: The information processing device performs the N+1th segmentation on the data to be processed after the Nth segmentation based on the target size threshold, until all data to be processed has been verified and stored.

[0093] Where N is a positive integer.

[0094] It should be noted that steps 304, 305-307 and step 308 are executed simultaneously.

[0095] Step 309: For each object's pending data, if the pending data has been stored and the pending data is determined to be valid based on the corresponding check value, the information processing device stores the metadata of the pending data.

[0096] Based on the foregoing embodiments, in other embodiments of this application, before step 306 where the information processing device determines the Nth verification request corresponding to the Nth sub-data of multiple objects from multiple verification requests stored in each request queue based on the position of the sub-data to be processed in the corresponding data to be processed, the information processing method further includes the following steps:

[0097] Step 310: Obtain processor usage information.

[0098] In this embodiment of the application, the usage of the processor on the information processing device can be detected.

[0099] Specifically, information processing equipment can determine the usage status of the processor based on its own operating status information.

[0100] In a possible implementation, the processor can be a central processing unit (CPU).

[0101] Step 311: determining the number of cores of the processor used for the verification processing based on the usage.

[0102] In the embodiments of the present application, the number of cores of the processor used for the verification processing can be determined from the multiple cores of the processor based on the usage, and the number of cores of the processor used for the verification processing can be counted.

[0103] Step 312: creating a request queue based on the number of cores, and storing each verification request into the request queue.

[0104] In the embodiments of the present application, the request queue can be configured based on the number of cores, wherein the number of cores is the same as the number of request queues, and the information processing device can store multiple verification requests into the request queue.

[0105] It should be noted that the storing of each verification request into the request queue in step 312 can be implemented by the following steps:

[0106] Step C1: for each verification request, determining the corresponding request queue based on the identity number of the verification request.

[0107] In the embodiments of the present application, the hash algorithm (Hash) can be used to determine the corresponding request queue of each verification request based on the identity number (Identity document, ID) of each verification request, wherein the verification requests of the same object are in the same request queue, wherein each request queue has multiple verification requests of objects, and the objects involved in different request queues are different.

[0108] Step 313: storing the verification request into the corresponding request queue.

[0109] In the embodiments of the present application, the generated verification request can be stored into the request queue corresponding to the verification request, so as to subsequently retrieve the verification request from the request queue to process the verification request.

[0110] It's important to note that during data validation, the validation of the current sub-data depends on the validation value of the previous sub-data and the current sub-data. This means that multiple sub-data items of the same object can only be validated sequentially, not simultaneously. Simultaneous validation will lead to validation failure. Therefore, validation requests for different sub-data items of the same object must be placed in the same request queue to prevent simultaneous validation of different sub-data items of the same object. Thus, for validation requests of different sub-data items of the same object, the ID can be set to be consistent. This ensures that sub-data items of the same object are assigned to the same request queue, preventing them from being assigned to different queues and causing simultaneous validation failure. Each sub-data item can be understood as a data fragment or data slice of the data to be processed. Each request queue is associated with a worker thread running on the CPU core of the instruction. The worker thread corresponding to the request queue performs batch processing on the validation requests in that queue, obtaining validation values ​​for multiple data slices of different objects.

[0111] In one feasible implementation, such as Figure 4 As shown, a hash algorithm can be used to perform a hash modulo operation on the ID of each verification request to determine the corresponding request queue. Verification requests are then stored in their respective queues. Based on the position of the data shard within the corresponding data to be processed, a batch of verification requests with distinct IDs are retrieved from each request queue. The worker thread corresponding to each request queue calls instructions from the advanced vector extension instruction set to process the verification requests retrieved from each queue, obtaining the verification value corresponding to the data shard carried in each request. This enables simultaneous verification of multiple objects' sub-data in a single request queue, as well as simultaneous batch verification of multiple objects' sub-data in multiple request queues. Thus, based on instructions from the architecture-specific advanced vector extension instruction set, batch execution of verification value calculations for data shards from multiple different objects is achieved, improving CPU utilization efficiency in the object storage system and reducing the time spent on verification processing. By hiding the verification processing and storage processing (i.e., storing data to be processed) involved in object storage, the time spent on object storage is shortened, ultimately solving the object upload latency problem. Figure 4 In this context, CPU1 refers to CPU core 1; CPUm refers to CPU core m.

[0112] It should be noted that through the advanced vector extension instruction set of the specific architecture, one CPU core can calculate the check values of multiple data fragments at a time, improving the efficiency of the CPU.

[0113] The following describes the information processing method provided by the embodiments of the present application in combination with the solutions in the related art.

[0114] As shown in the information processing method provided by the embodiments of the present application, the server first reads the user data uploaded by the user, then calculates the check value of the user data, and then writes the user data into the storage pool, and finally stores the metadata. Figure 5 It can be seen that reading the user data, checking the user data, and writing the user data into the storage pool are serial processes, which will cause a relatively high object upload delay. Based on this, the embodiments of the present application provide an information processing method, which is applied to an information processing system, as shown in the information processing method provided by the embodiments of the present application, the server first reads the user data uploaded by the user, then calculates the check value of the user data, and then writes the user data into the storage pool, and finally stores the metadata. Figure 6 As shown in the information processing method provided by the embodiments of the present application, the server first reads the user data uploaded by the user, then calculates the check value of the user data, and then writes the user data into the storage pool, and finally stores the metadata.

[0115] In the embodiments of the present application, the data reading module is configured to read the to-be-processed data from a storage request and transmit the to-be-processed data to the data stream splitting module. The data stream splitting module can continuously split the to-be-processed data based on a target size threshold. When splitting, the data stream splitting module does not need to wait for the current to-be-processed sub-data to be processed and then perform the next splitting, but performs the next splitting immediately after the current to-be-processed sub-data (data fragment) is obtained. In this way, the time spent on splitting the to-be-processed data is reduced. The data stream splitting module can trigger asynchronous storage and asynchronous checking of the to-be-processed sub-data. Specifically, the data stream splitting module can write the data fragment split by the storage pool access agent module into the storage pool, and simultaneously check the data fragment by the MD5 check module. That is, when a data fragment is stored, the data fragment is also checked at the same time. In this way, the checking and storage of the data fragment are parallel processed. When the data fragment is checked and stored, the data stream splitting module can continue to split the remaining to-be-processed data, so that different data fragments of the same object are processed in an asynchronous and concurrent manner. The multiple time-consuming stages of object processing are hidden from each other, so that the object upload delay is shortened.

[0116] It should be noted that the data to be processed is divided into multiple smaller granularity data segments by the data stream segmentation module, the data verification process is asynchronous, without waiting for the previous data segment to be written into the backend storage pool to start cutting the next segment, realizing parallel processing of the read user data stage, data verification calculation stage, and writing into the backend storage pool stage, thereby shortening the overall object upload time and solving the problem of long object upload time. (The object upload can be understood as object storage). Through product environment Zhang verification, under the premise that the storage pool and CPU are not bottlenecked, the object average upload delay can be reduced by about 35%.

[0117] It should be noted that the descriptions of the same steps and contents in this embodiment and other embodiments can refer to the descriptions in other embodiments, which will not be repeated here.

[0118] The information processing method provided by the embodiments of the present application can verify the to-be-processed data of multiple objects in parallel through the execution of the multi-data operation instruction, realize the simultaneous batch verification of multiple to-be-processed data, reduce the time spent on object storage of massive data, improve the efficiency of object storage, and solve the problem of long time and low efficiency in related technologies when facing massive data and performing object storage.

[0119] Based on the foregoing embodiments, the embodiments of the present application provide an information processing device, which can be applied to Figures 1-3 The information processing method provided by the corresponding embodiments can refer to Figure 7 As shown in the figure, the information processing device 4 can include:

[0120] The acquisition unit 41 is configured to acquire the to-be-processed data of multiple objects.

[0121] The processing unit 42 is configured to verify the to-be-processed data of multiple objects in parallel through the target instruction, obtain the verification value corresponding to each to-be-processed data, and store the to-be-processed data of each object; wherein the target instruction is an instruction for performing multi-data operation.

[0122] The processing unit 42 is further configured to, for the to-be-processed data of each object, in a case where the to-be-processed data has been stored and the to-be-processed data is determined to be legal based on the verification value corresponding to the to-be-processed data, store the metadata of the to-be-processed data.

[0123] In the embodiments of the present application, the acquisition unit 41 is further configured to perform the following steps:

[0124] Receive multiple processing requests; the multiple processing requests are used to request to store the to-be-processed data of multiple objects.

[0125] Obtain the to-be-processed data of multiple objects from the multiple processing requests.

[0126] In the embodiment of the present application, the processing unit 42 is further configured to perform the following steps:

[0127] For the to-be-processed data of each object, the to-be-processed data is segmented based on the target size threshold value, in the case that each segmentation obtains to-be-processed sub-data, a verification value is obtained by verifying the to-be-processed sub-data of each object through the target instruction, and the to-be-processed sub-data is stored.

[0128] In the embodiment of the present application, the processing unit 42 is further configured to perform the following steps:

[0129] For the to-be-processed data of each object, the to-be-processed data is segmented for the Nth time based on the target size threshold value, to obtain the Nth to-be-processed sub-data;

[0130] The Nth to-be-processed sub-data is stored and verified through the target instruction, and the to-be-processed data after the Nth segmentation is segmented for the (N+1)th time based on the target size threshold value, until the to-be-processed data is verified and stored completely; wherein N is a positive integer.

[0131] In the embodiment of the present application, the processing unit 42 is further configured to perform the following steps:

[0132] A verification request for the to-be-processed sub-data of each object is generated; wherein the verification request carries the to-be-processed sub-data;

[0133] Based on the position of the to-be-processed sub-data in the corresponding to-be-processed data, the Nth verification request corresponding to the Nth to-be-processed sub-data of the plurality of objects is determined from the plurality of verification requests stored in each request queue;

[0134] Based on the Nth verification request of the plurality of objects in each request queue, the Nth to-be-processed sub-data carried in the Nth verification request of the plurality of objects in each request queue is verified through the target instruction at the same time, to obtain the Nth verification value corresponding to each Nth to-be-processed sub-data.

[0135] In the embodiment of the present application, the processing unit 42 is further configured to perform the following steps:

[0136] The usage of the processor is obtained, and the number of cores of the processor used for verification processing is determined based on the usage;

[0137] The request queue is created based on the number of cores, and each verification request is stored in the request queue.

[0138] In the embodiment of the present application, the processing unit 42 is further configured to perform the following steps:

[0139] For each verification request, based on the identity number of the verification request, a corresponding request queue is determined;

[0140] The verification request is stored in the corresponding verification request; wherein a plurality of verification requests corresponding to the same object are stored in the same request queue.

[0141] In the embodiments of the present application, the processing unit 42 is further configured to perform the following steps:

[0142] Through the target function, a vector extension instruction set is called;

[0143] Through the instructions in the vector extension instruction set, the Nth to-be-processed sub-data carried in the plurality of Nth verification requests of the plurality of objects in each request queue is simultaneously subjected to integrity verification, and the Nth verification value corresponding to each Nth to-be-processed sub-data is obtained.

[0144] It should be noted that the interaction process between the units in the embodiments of the present application can refer to the implementation process in the information processing method provided by the corresponding embodiments, which will not be described here in detail. Figures 1-3 The implementation process in the information processing method provided by the corresponding embodiments, which will not be described here in detail.

[0145] The information processing device provided by the embodiments of the present application can perform verification on the to-be-processed data of a plurality of objects in parallel through instructions for performing multi-data operations when facing massive data, thereby realizing simultaneous batch verification of a plurality of to-be-processed data, reducing the time spent when storing objects for massive data, improving the efficiency of object storage, and solving the problem of long time consumption and low efficiency when storing objects for massive data in related technologies.

[0146] Based on the foregoing embodiments, the embodiments of the present application provide an information processing device, which can be applied to Figures 1-3 In the information processing method provided by the corresponding embodiments, refer to Figure 8 As shown in the figure, the information processing device 5 can include a processor 51, a memory 52 and a communication bus 53, wherein:

[0147] The communication bus 53 is configured to realize the communication connection between the processor 51 and the memory 52;

[0148] The processor 51 is configured to execute the information processing program stored in the memory 52 to realize the following steps:

[0149] Obtain the to-be-processed data of a plurality of objects;

[0150] Verify the to-be-processed data of a plurality of objects in parallel through target instructions to obtain the verification value corresponding to each to-be-processed data, and store the to-be-processed data of each object; wherein the target instruction is an instruction for performing multi-data operation;

[0151] In the case that the to-be-processed data of each object is stored and the to-be-processed data is determined to be legal based on the corresponding check value of the to-be-processed data, the metadata of the to-be-processed data is stored.

[0152] In other embodiments of the present application, the processor 51 is configured to execute the information processing program stored in the memory 52 to acquire the to-be-processed data of the plurality of objects, so as to implement the following steps:

[0153] receiving a plurality of processing requests; the plurality of processing requests are used to request to store the to-be-processed data of the plurality of objects;

[0154] acquiring the to-be-processed data of the plurality of objects from the plurality of processing requests.

[0155] In other embodiments of the present application, the processor 51 is configured to execute the information processing program stored in the memory 52 to perform the check on the to-be-processed data of the plurality of objects in parallel through the target instruction, to obtain the corresponding check value of each to-be-processed data, and to store the to-be-processed data of each object, so as to implement the following steps:

[0156] In the case that the to-be-processed data of each object is stored and the to-be-processed data is determined to be legal based on the corresponding check value of the to-be-processed data, the metadata of the to-be-processed data is stored.

[0157] In other embodiments of the present application, the processor 51 is configured to execute the information processing program stored in the memory 52 to perform the check on the to-be-processed data of the plurality of objects in parallel through the target instruction, to obtain the corresponding check value of each to-be-processed data, and to store the to-be-processed data of each object, so as to implement the following steps:

[0158] In the case that the to-be-processed data of each object is stored and the to-be-processed data is determined to be legal based on the corresponding check value of the to-be-processed data, the metadata of the to-be-processed data is stored.

[0159] In the case that the to-be-processed data of each object is stored and the to-be-processed data is determined to be legal based on the corresponding check value of the to-be-processed data, the metadata of the to-be-processed data is stored.

[0160] In other embodiments of the present application, the processor 51 is configured to execute the information processing program stored in the memory 52 to perform the check on the to-be-processed data of the plurality of objects in parallel through the target instruction, to obtain the corresponding check value of each to-be-processed data, and to store the to-be-processed data of each object, so as to implement the following steps:

[0161] generating a check request for each object for the to-be-processed sub-data; wherein the check request carries the to-be-processed sub-data;

[0162] determining, based on the position of the to-be-processed sub-data in the corresponding to-be-processed data, the Nth check request corresponding to the Nth to-be-processed sub-data of the plurality of objects from the plurality of check requests stored in each request queue;

[0163] based on the plurality of Nth check requests of the plurality of objects in each request queue, performing integrity check on the Nth to-be-processed sub-data carried in the plurality of Nth check requests of the plurality of objects in each request queue simultaneously through the target instruction, to obtain the Nth check value corresponding to each Nth to-be-processed sub-data.

[0164] In other embodiments of the present application, the processor 51 is configured to execute the information processing program stored in the memory 52 to implement the following steps:

[0165] obtaining the usage of the processor, and determining the number of cores of the processor for check processing based on the usage;

[0166] creating a request queue based on the number of cores, and storing each check request to the request queue.

[0167] In other embodiments of the present application, the processor 51 is configured to store each check request to the request queue by executing the information processing program stored in the memory 52, to implement the following steps:

[0168] for each check request, determining the corresponding request queue based on the identity number of the check request;

[0169] storing the check request to the corresponding check request; wherein the plurality of check requests corresponding to the same object are stored in the same request queue.

[0170] In other embodiments of the present application, the processor 51 is configured to execute the information processing program stored in the memory 52 to implement the following steps:

[0171] calling a vector extension instruction set through the target function;

[0172] performing integrity check on the Nth to-be-processed sub-data carried in the plurality of Nth check requests of the plurality of objects in each request queue simultaneously through the instructions in the vector extension instruction set, to obtain the Nth check value corresponding to each Nth to-be-processed sub-data.

[0173] It should be noted that the specific implementation process of the steps performed by the processor in this embodiment can be referred to Figures 1-3 The implementation process in the information processing method provided by the corresponding embodiment is not described here again.

[0174] The information processing device provided by the embodiment of the present application can verify the to-be-processed data of multiple objects in parallel by executing the instruction of multi-data operation when facing massive data, thereby realizing the simultaneous batch verification of multiple to-be-processed data, reducing the time spent when object storage is performed on massive data, improving the efficiency of object storage, and solving the problem of long time consumption and low efficiency when object storage is performed on massive data in the related art.

[0175] Based on the foregoing embodiments, the embodiment of the present application provides a computer readable storage medium, which stores one or more programs executable by one or more processors to implement Figures 1-3 The steps in the information processing method provided by the corresponding embodiment.

[0176] It should be noted that the computer readable storage medium described above can be a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a Ferromagnetic Random Access Memory (FRAM), a Flash Memory, a magnetic surface memory, an optical disc, or a Compact Disc Read-Only Memory (CD-ROM) memory; or can be various electronic devices including one or any combination of the above memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc.

[0177] It should be noted that, in the present document, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0178] The above-mentioned sequence numbers of embodiments of the present application are only for description, and do not represent advantages or disadvantages of the embodiments.

[0179] Those skilled in the art can clearly understand the above-mentioned embodiment method by means of software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, or optical disc) and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device) to execute the methods described in various embodiments of the present application.

[0180] The present application is described with reference to the flowcharts and / or block diagrams according to the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0181] These computer program instructions can also be stored in a computer-readable storage medium that can direct the computer or other programmable data processing devices to work in a specific manner, so that the instructions stored in the computer-readable storage medium produce a manufactured product including instruction devices that implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0182] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate a computer implemented process, so that the instructions executed on the computer or other programmable data processing devices provide a process for implementing the functions specified in the flowchart Figure 1 one flow or multiple flows and / or the functions specified in the block Figure 1 one flow or multiple flows and / or the functions specified in the block

[0183] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent flow transformation using the content of the specification and drawings of the present application, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An information processing method, characterized in that, The method includes: Retrieve pending data from multiple objects; For each object's data to be processed, the data to be processed is divided into Nth segments based on the target size threshold to obtain the Nth sub-data to be processed; The process involves storing the Nth sub-data to be processed and verifying it using a target instruction. Simultaneously, based on the target size threshold, the data to be processed after the Nth segmentation is divided for the (N+1)th time, until all data to be processed has been verified and stored. Here, N is a positive integer; the target instruction is an instruction that performs multi-data operations. For each object's pending data, if the pending data has been stored and the pending data is determined to be valid based on the corresponding check value, the metadata of the pending data is stored. The step of verifying the Nth sub-data to be processed through the target instruction includes: Generate a verification request for the sub-data to be processed for each object; wherein the verification request carries the sub-data to be processed; Based on the position of the sub-data to be processed in the corresponding data to be processed, determine the Nth verification request corresponding to the Nth sub-data to be processed for multiple objects from the multiple verification requests stored in each request queue; Based on multiple Nth verification requests of the multiple objects in each request queue, the target instruction simultaneously verifies the Nth sub-data to be processed carried in the multiple Nth verification requests of the multiple objects in each request queue, and obtains the Nth verification value corresponding to each Nth sub-data to be processed.

2. The method according to claim 1, characterized in that, The process of acquiring data to be processed from multiple objects includes: Receive multiple processing requests, the multiple processing requests being used to request the storage of pending data of the multiple objects; Obtain the pending data of the multiple objects from the multiple processing requests.

3. The method according to claim 1, characterized in that, Before determining the Nth verification request corresponding to the Nth sub-data of multiple objects from multiple verification requests stored in each request queue based on the position of the sub-data to be processed in the corresponding data to be processed, the method further includes: Get processor usage information; The number of processor cores used for verification processing is determined based on the aforementioned usage. A request queue is created based on the number of kernels, and each verification request is stored in the request queue.

4. The method according to claim 3, characterized in that, The step of storing each verification request to the request queue includes: For each verification request, the corresponding request queue is determined based on the identity identifier of the verification request; Verification requests are stored in the corresponding request queue; multiple verification requests for the same object are stored in the same request queue.

5. The method according to claim 1, characterized in that, The step of simultaneously verifying the Nth unprocessed sub-data carried in multiple Nth verification requests of multiple objects in each request queue through the target instruction to obtain the Nth verification value corresponding to each Nth unprocessed sub-data includes: The target function invokes the vector extended instruction set; Using the instructions in the vector extended instruction set, the integrity of the Nth sub-data to be processed carried in the multiple Nth verification requests of the multiple objects in each request queue is simultaneously verified to obtain the Nth verification value corresponding to each Nth sub-data to be processed.

6. An information processing device, characterized in that, The device includes: The acquisition unit is used to acquire data to be processed from multiple objects. The processing unit is configured to, for each object's data to be processed, perform an Nth segmentation based on a target size threshold to obtain an Nth sub-data to be processed; store the Nth sub-data to be processed and verify the Nth sub-data to be processed by a target instruction; simultaneously, perform an N+1th segmentation on the data to be processed after the Nth segmentation based on the target size threshold, until all data to be processed has been verified and stored; where N is a positive integer; the target instruction is an instruction for performing multi-data operations; The processing unit is further configured to generate a verification request for each object's unprocessed sub-data; wherein the verification request carries the unprocessed sub-data; based on the position of the unprocessed sub-data in the corresponding unprocessed data, determine the Nth verification request corresponding to the Nth unprocessed sub-data of multiple objects from multiple verification requests stored in each request queue; based on the multiple Nth verification requests of the multiple objects in each request queue, simultaneously verify the Nth unprocessed sub-data carried in the multiple Nth verification requests of the multiple objects in each request queue through the target instruction, and obtain the Nth verification value corresponding to each Nth unprocessed sub-data; The processing unit is further configured to, for each object's data to be processed, store the metadata of the data to be processed, provided that the data to be processed has been stored and the data to be processed is determined to be valid based on the verification value corresponding to the data to be processed.

7. An information processing device, characterized in that, The device includes: a processor, a memory, and a communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is used to execute the information processing program stored in the memory to implement the steps of the information processing method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the information processing method as described in any one of claims 1 to 5.

Images