Data processing method of shared memory, related device, product and medium

Abstract

The embodiment of the application discloses a data processing method of a shared memory, related equipment, products and media, which can be applied to the technical field of data processing. The method comprises the following steps: receiving a data processing request initiated based on a computing task of a distributed application, and detecting a request result of the data processing request in a local memory; a plurality of devices are used for processing the computing task of the distributed application; if the request result of the data processing request is not detected in the local memory, the request result of the data processing request is detected in a shared memory; if the request result of the data processing request is not detected in the shared memory, the request result of the data processing request is calculated in the local memory, and the calculated request result is added to the shared memory. By adopting the embodiment of the application, the processing efficiency of the computing task of the distributed application can be improved. The embodiment of the application can also be applied to various scenes such as cloud technology, artificial intelligence, intelligent transportation and auxiliary driving.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, related equipment, products and media for shared memory. Background Technology

[0002] A distributed application refers to an application distributed across different computers, working together over a network to complete a task. Such a distributed application can utilize multiple data processing devices to handle the same computational task. In existing technologies, any data processing device in a distributed application, upon receiving a data processing request for that computational task, can locally compute the corresponding request result. However, the inventors discovered in practice that using this computational method in distributed applications can lead to the same computational operations being performed on different computer devices, resulting in redundant data computation and reduced task processing efficiency. Summary of the Invention

[0003] This application provides a data processing method, related equipment, products, and media for shared memory, which helps to improve the processing efficiency of computing tasks in distributed applications.

[0004] On one hand, embodiments of this application disclose a data processing method for a shared memory, the method comprising:

[0005] It receives data processing requests initiated by computing tasks based on distributed applications and detects the request results in local storage; multiple devices are used to process computing tasks of distributed applications.

[0006] If the result of the data processing request is not detected in local memory, the result of the data processing request is detected in shared memory.

[0007] If no result for the data processing request is detected in the shared memory, the result for the data processing request is calculated in the local memory and then added to the shared memory.

[0008] On one hand, embodiments of this application disclose a data processing apparatus for a shared memory, the apparatus comprising:

[0009] A receiving unit is used to receive data processing requests initiated by computing tasks based on distributed applications and to detect the request results of the data processing requests in local memory; multiple devices are used to process computing tasks of distributed applications.

[0010] The processing unit is configured to detect the result of the data processing request in the shared memory if the request result of the data processing request is not detected in the local memory.

[0011] The processing unit is further configured to, if no request result for the data processing request is detected in the shared memory, calculate the request result for the data processing request in the local memory and add the calculated request result to the shared memory.

[0012] On one hand, embodiments of this application provide an electronic device, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to perform the following steps:

[0013] It receives data processing requests initiated by computing tasks based on distributed applications and detects the request results in local storage; multiple devices are used to process computing tasks of distributed applications.

[0014] If the result of the data processing request is not detected in local memory, the result of the data processing request is detected in shared memory.

[0015] If no result for the data processing request is detected in the shared memory, the result for the data processing request is calculated in the local memory and then added to the shared memory.

[0016] On one hand, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, perform the following steps:

[0017] It receives data processing requests initiated by computing tasks based on distributed applications and detects the request results in local storage; multiple devices are used to process computing tasks of distributed applications.

[0018] If the result of the data processing request is not detected in local memory, the result of the data processing request is detected in shared memory.

[0019] If no result for the data processing request is detected in the shared memory, the result for the data processing request is calculated in the local memory and then added to the shared memory.

[0020] On one hand, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative embodiments described above.

[0021] This application proposes a data processing scheme for shared memory, applicable to any one of multiple devices connected to the shared memory. Upon receiving a data processing request from a computing task for a distributed application, the scheme first checks the request result in local memory. If no corresponding request result is found in local memory, it checks the request result in shared memory. If no corresponding request result is found in shared memory, the request result is calculated in local memory and then added to the shared memory. Since any device connected to the shared memory calculates and uploads its request result to the shared memory, subsequent data processing requests can be processed by sequentially checking both local and shared memory for request results. This reduces performance overhead caused by redundant operations on multiple devices connected to the shared memory, thereby improving the processing efficiency of distributed application computing tasks. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram illustrating the effect of a shared memory data processing system provided in an embodiment of this application;

[0024] Figure 2 This is a schematic diagram of the structure of a data processing device for a shared memory provided in an embodiment of this application;

[0025] Figure 3 This is a schematic flowchart of a data processing method for a shared memory provided in an embodiment of this application;

[0026] Figure 4 This is a schematic flowchart of a data processing method for a shared memory provided in an embodiment of this application;

[0027] Figure 5 This is a schematic diagram of the framework of a data processing method for a shared memory provided in an embodiment of this application;

[0028] Figure 6 This is a flowchart illustrating a method for obtaining request results provided in an embodiment of this application;

[0029] Figure 7 This is a schematic diagram of the structure of a data processing device for a shared memory provided in an embodiment of this application;

[0030] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0031] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0032] This application proposes a data processing scheme for shared memory, applicable to any one of multiple devices connected to the shared memory. Upon receiving a data processing request from a computing task for a distributed application, the scheme first checks the request result in local memory. If no corresponding request result is found in local memory, it checks the request result in shared memory. If no corresponding request result is found in shared memory, the request result is calculated in local memory and then added to the shared memory. Since any device connected to the shared memory calculates and uploads its request result to the shared memory, subsequent data processing requests can be processed by sequentially checking both local and shared memory for request results. This reduces performance overhead caused by redundant operations on multiple devices connected to the shared memory, thereby improving the processing efficiency of distributed application computing tasks.

[0033] In one possible implementation, the above-described shared memory data processing scheme can be applied to more complex data processing systems. See also... Figure 1 , Figure 1 This is a schematic diagram illustrating the effect of a data processing system provided in an embodiment of this application. For example... Figure 1 As shown, the data processing system can be a distributed application system, which may include multiple data processing devices, request initiators, and a cloud storage system. The data processing device refers to a device used to process the computing tasks of the distributed application, and this data processing device contains an application program for processing these computing tasks.

[0034] Each of the at least one data processing device includes corresponding local memory, and each of the at least one data processing device is connected to the same shared memory. That is, each of the at least one data processing device includes both local memory and shared memory. It is understood that, for shared memory, multiple devices (i.e., the aforementioned multiple data processing devices) are connected to it. The data processing scheme for the shared memory is applied to any one of the multiple devices connected to the shared memory, and all the multiple data processing devices connected to the shared memory can be referred to as the connected devices of the shared memory. These multiple devices are used to process the computing tasks of distributed applications. Please see [link to relevant documentation]. Figure 2 , Figure 2 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. For example... Figure 2 As shown, the data processing device 1 may include a corresponding local memory (such as...). Figure 2 As shown in 201), and is connected to shared memory (such as...). Figure 2 As shown in 202); the data processing device 2 may include a corresponding local memory (such as...). Figure 2 As shown in 203), and also connected to, as shown in, Figure 2 The shared memory shown in 202; similarly, the data processing device 3 may include a corresponding local memory (such as...). Figure 2 As shown in 204), and also connected to, as shown in, Figure 2 The shared memory is shown in 202.

[0035] In this system, local storage can be a local disk, referring to a disk (hard drive) installed on the same computer motherboard that cannot be freely plugged in or moved. Shared storage can be a shared disk, referring to storage that can be shared by connected devices, meaning that any connected device can read and write data to this shared storage. Any data processing device among these connected devices can be configured with the aforementioned shared storage data processing scheme. Upon receiving a data processing request from the request initiator, it sequentially checks whether the request result exists in both local and shared storage. If neither local nor shared storage contains the request result, the request result can be calculated based on local storage and added to shared storage. This allows connected devices to retrieve the corresponding request result from shared storage. Therefore, once any data processing device in the data processing system calculates the request result, other data processing devices do not need to perform recalculation; they can directly obtain the request result from the shared disk. This ensures data consistency in the distributed application, reduces performance losses caused by invalid duplicate operations, and improves the processing efficiency of distributed application computing tasks. Furthermore, this application leverages the superior read / write performance of local memory compared to shared memory, prioritizing the reading of data (i.e., the requested result here) from local memory. This results in higher data read / write efficiency. If the required data is not present in local memory, it is then read from shared memory, which helps improve operational performance and thus enhances the processing efficiency of the computational task. In addition, since this application computes the requested result in local memory, it avoids the slow read / write operations of shared memory, further contributing to improved data processing efficiency.

[0036] This request initiator can be used to initiate data processing requests for computing tasks in distributed applications. These requests instruct the corresponding data processing operations. The cloud storage system can be used to store source files required by these data processing requests. Understandably, the processing of data processing requests involves data read and write operations. A read operation indicates the act of reading data from the storage area of ​​the data processing device (such as the shared memory or local memory mentioned above), while a write operation indicates the act of writing data into the storage area of ​​the data processing device, also known as adding data to the storage area.

[0037] For example, a data processing request initiated for a computing task in a distributed application instructs the acquisition of coverage between compiled file A and executable file B. This coverage describes the proportion and extent to which the source code in the program has been tested; that is, it is determined by which code has been executed and which has not. If this data processing request is routed to device a (i.e., the data processing request is assigned to device a for processing), after receiving the data processing request, device a sequentially checks whether the request result exists in local memory and shared memory. If the request result is not found in shared memory, device a pulls compiled file A and executable file B from the cloud storage system, writes compiled file A and executable file B to local memory, and then reads compiled file A and executable file B from local memory to calculate the coverage of compiled file A and executable file B, obtaining the requested result (i.e., result file C). Result file C is then written to local memory and added to shared memory. After being stored in shared memory, result file C can be retrieved by other devices in the data processing system besides device a.

[0038] It should be noted that this application may display a prompt interface or pop-up window before collecting user data (such as user data that requires data processing) and during the collection process. This prompt interface or pop-up window is used to inform the user that their data is being collected. This application will only begin the steps related to collecting user data after receiving confirmation from the user regarding the prompt interface or pop-up window; otherwise (i.e., without receiving confirmation from the user), the steps to collect user data will end, meaning no user data will be collected. In other words, all user data collected in this application is collected with the user's consent and authorization, and the collection, use, and processing of related user data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0039] The technical solution of this application can be applied to electronic devices, such as any one of the multiple devices connected to shared memory as described above. The electronic device can be a terminal, a server, or other devices used for data processing; this application does not limit the scope. Optionally, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminals include, but are not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, and aircraft.

[0040] It is understood that the above scenarios are merely examples and do not constitute a limitation on the application scenarios of the technical solutions provided in the embodiments of this application. The technical solutions of this application can also be applied to other scenarios. For example, as those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0041] Based on the above description, this application proposes a data processing method for shared memory. Please refer to... Figure 3 , Figure 3 This is a schematic flowchart illustrating a data processing method for shared memory provided in an embodiment of this application. The method can be executed by the aforementioned electronic device, which includes local memory and shared memory. The data processing method for shared memory may include the following steps.

[0042] S301. Receive a data processing request initiated by a computing task based on a distributed application, and detect the request result of the data processing request in the local storage.

[0043] The data processing request can be initiated by any computing task based on a distributed application. A computing task in a distributed application can be broken down into multiple subtasks, each of which can be processed by different data processing devices. For any given data processing device, it can receive a data processing request instructing the completion of a subtask of the computing task; that is, a data processing request initiated based on that computing task. For example, a computing task can be broken down into subtask 1, subtask 2, and subtask 3. The target device can receive a data processing request instructing the completion of subtask 1. This request could be used to instruct the acquisition of the coverage ratio between compiled file A and executable file B. Subsequently, the computing task of the distributed application can be processed based on the request results corresponding to each subtask.

[0044] In one possible implementation, the data processing request can be initiated by a corresponding requesting device, and then the data processing request is routed to a target device. The target device can be any of the multiple devices connected to the shared memory, and the target device can receive the data processing request to process it. As mentioned above, the requesting device is the request initiator in the data processing system. The requesting device can be an application client or an application backend server; there is no limitation here.

[0045] In one possible implementation, detecting the request result of a data processing request in local memory may specifically include the following steps: ① generating a request identifier for the data processing request; ② detecting the request result corresponding to the data processing request in shared memory based on the request identifier. The request identifier can be used to identify the request result corresponding to the data processing request. It is understood that before receiving the data processing request, the device may have calculated the corresponding request result in response to other data processing requests and associated the request result with the request identifier identifying the request result in local memory. That is, if there is data in local memory where the associated request identifier is the same as the result identifier of the data processing request, it indicates that the device has previously calculated the request result of the data processing request, and thus the request result corresponding to the data processing request can be detected in local memory. If there is no data in local memory where the associated request identifier is the same as the result identifier of the data processing request, it indicates that the device has not previously calculated the request result of the data processing request, and thus the request result corresponding to the data processing request cannot be detected in local memory.

[0046] In one possible implementation, the request identifier can be determined based on key request parameters in the data processing request. For example, the corresponding request identifier can be obtained by hashing the key request parameters. These key request parameters refer to parameters involved in calculating the request result. Parameters not involved in calculating the request result can be excluded from determining the request identifier, such as timestamp parameters not involved in the request result calculation, device identifiers of the request initiating device, etc., which are not considered key request parameters. For instance, if a data processing request initiated for a distributed application's computing task indicates the acquisition of coverage between compiled file A and executable file B, the relevant parameters indicating the calculation of coverage between compiled file A and executable file B can be extracted and hashed to determine the request identifier.

[0047] S302. If the request result of the data processing request is not detected in the local memory, then the request result of the data processing request is detected in the shared memory.

[0048] As mentioned above, if the request result for the data processing request is not detected in the local memory, it means that the device has not calculated the request result in the past, or the calculated result was not written to the local memory, or the request result stored in the local memory was deleted, etc. Therefore, the request result for the data processing request is detected in the shared memory, that is, whether other devices connected to the shared memory have calculated the request result for the data processing request. This leverages the superior read / write performance of local memory compared to shared memory. If the required data is in the local memory, it is read first (i.e., the request result in this case), resulting in higher data read / write efficiency. If the required data is not in the local memory, it is then read from the shared memory, which helps improve operational performance and thus enhances the processing efficiency of computing tasks. Furthermore, since this application can first detect whether there is a request result for a data processing request in the local memory, instead of directly detecting whether there is a request result in the shared memory as is usually the case, it can better avoid duplicate calculations. For example, due to the slow nature of writing data to shared memory, after a device calculates the request result for a data processing request, it is easy for the request result to be not completely added to the shared memory before receiving another data processing request that needs to obtain the request result. At this time, the device cannot read the request result from the shared memory and thus recalculates the request result for the data processing request, resulting in duplicate calculation of the request result. By first detecting whether there is a request result in the local memory through the embodiments of this application, the performance loss caused by invalid duplicate operations is reduced, and the processing efficiency of computing tasks in distributed applications is improved.

[0049] In one possible implementation, detecting the request result of a data processing request in shared memory may specifically include the following steps: detecting the request result corresponding to the data processing request in shared memory based on the request identifier. As described above, the relevant description of the request identifier can be referred to the above description, and will not be repeated here. That is to say, before detecting the request result corresponding to the data processing request in shared memory based on the request identifier, it is necessary to first generate the request identifier of the data processing request.

[0050] Understandably, if, before receiving the data processing request, any other device connected to the shared memory (excluding the device itself) has calculated the request result corresponding to the data processing request in response to other data processing requests, and added the request result along with a request identifier identifying the request result to the shared memory, then if there is data in the shared memory where the associated request identifier is the same as the result identifier of the data processing request, it means that any other device connected to the shared memory (excluding the device itself) has calculated the request result of the data processing request, and thus the request result corresponding to the data processing request can be detected in the shared memory. If there is no data in the shared memory where the associated request identifier is the same as the result identifier of the data processing request, it means that no other device connected to the shared memory (excluding the device itself) has calculated the request result of the data processing request in the past, and thus the request result corresponding to the data processing request cannot be detected in the shared memory.

[0051] In one possible implementation, if a request result for a data processing request is detected in local memory, the request result is retrieved from local memory; the computational task of the distributed application is then processed based on the retrieved request result. As mentioned above, the data processing request can be the request result corresponding to a subtask within the computational task. Therefore, processing the computational task of the distributed application based on the retrieved request result can be achieved by combining the request result of the data processing request with the request results of other subtasks. This allows the device to directly retrieve the request result from local memory after it has already calculated it, avoiding redundant calculations and reducing wasted computing resources and device performance degradation. This results in higher processing efficiency for data processing requests. If the above operation is performed on the data processing requests corresponding to each subtask of the distributed application's computational task, the processing efficiency of the distributed application's computational task can be significantly improved.

[0052] S303. If the request result of the data processing request is not detected in the shared memory, the request result of the data processing request is calculated in the local memory and added to the shared memory.

[0053] As mentioned above, if no result for a data processing request is detected in the shared memory, it means that none of the other devices connected to the shared memory have previously calculated the result for that data processing request, or the result stored in the shared memory has been deleted, etc. Therefore, before calculating the result for a data processing request, it's possible to check whether any other devices connected to the shared memory have the desired result. This avoids duplicate calculations of the data processing request result, reduces waste of computing resources and device performance degradation, and improves the processing efficiency of distributed application computing tasks.

[0054] In one possible implementation, if a request result for a data processing request is detected in the shared memory, the request result is retrieved from the shared memory; the computational task of the distributed application is then processed based on the request result retrieved from the shared memory. The process of processing the computational task of the distributed application based on the request result retrieved from the shared memory can be referred to the above description and will not be repeated here. Therefore, after the request result for the data processing request has already been calculated in any data processing device of the distributed application (i.e., any device connected to the shared memory), the request result can be directly retrieved from the shared memory. This avoids the repeated calculation of the request result by multiple data processing devices in the distributed application, reducing the waste of computing resources and the performance loss of devices, thereby improving the processing efficiency of data processing requests. If the above operation is performed on the data processing request corresponding to each subtask of the distributed application's computational task, the processing efficiency of the distributed application's computational task can be greatly improved.

[0055] The process of calculating the request result corresponding to the data processing request in local storage refers to reading the file required for data processing from local storage, processing it based on local computing resources to obtain the corresponding request result, and then writing the request result back to local storage.

[0056] It is understandable that the device contains an application for processing data processing requests and obtaining corresponding results, i.e., the application mentioned above for handling computational tasks. Therefore, after receiving a data processing request, the target device can calculate the corresponding target request result based on the application corresponding to the data processing request. For example, a data processing request could be used to instruct the acquisition of the coverage ratio between compiled file A and executable file B. The application corresponding to the data processing request can then calculate the coverage ratio between compiled file A and executable file B to obtain the target request result.

[0057] It is understandable that if the request result for the data processing request does not exist in the shared memory, it means that none of the connected devices in the shared memory have calculated the request result before obtaining the data processing request. In this case, the target device needs to perform the calculation in the local memory to obtain the target request result.

[0058] In one possible implementation, after calculating the request result of the data processing request, the computing tasks of the distributed application can be processed based on the calculated request result.

[0059] In one possible implementation, some intermediate data generated during the data processing of the application can be stored in a database, also known as the business database.

[0060] It is understandable that after the computational request result is added to the shared memory, this result can be retrieved by multiple devices connected to the shared memory. Therefore, once any device connected to the shared memory adds its request result, it can retrieve the result from the shared memory whenever it needs to obtain it, greatly reducing the waste of computing resources and minimizing the waste of local memory space for each connected device. Specifically, if any device needs to retrieve the target request result from the shared memory, it means that after receiving a data processing request that requires the result, the request result is not detected in the device's local memory. In other words, after receiving a data processing request, any device can follow the steps described above: first, check the request result in its local memory; if the result is not detected in its local memory, then check the request result in the shared memory. Since the computational result was added to the shared memory during this process, any device can retrieve the result from the shared memory.

[0061] For example, the devices connected to the shared memory include devices a, b, and c. Device a receives a data processing request with a request identifier marked T1. If device a detects that neither the local memory nor the shared memory contains data associated with the request identifier T1, it means that before receiving the data processing request, devices a, b, and c had not calculated the request result corresponding to T1. Therefore, device a can calculate the request result of the data processing request in the local memory and associate the calculated request result with the request identifier T1 in both the local memory and the shared memory. After this, if any of devices a, b, and c receives a data processing request with the request identifier T1, device a can detect the data associated with T1 in its local memory and retrieve it as the request result of the newly received data processing request. For devices b and c, no data associated with T1 is detected in their local memory, so they can detect it in the shared memory and retrieve it as the request result of the newly received data processing request. Therefore, after any device connected to the shared memory has calculated the request result of the data processing request, any subsequent device that needs to obtain the request result can directly retrieve it from the local memory or the shared memory without further calculation, greatly improving data processing efficiency and reducing performance loss caused by repeated calculations.

[0062] In one possible implementation, when adding the request result of a data processing request to shared storage, a soft link can be used. This can include the following steps: uploading the request result to a temporary directory in the shared storage; and creating a soft link corresponding to the request result, pointing to the request result in the temporary directory. The temporary directory is a place where the system temporarily stores files that only need to be used once or a few times during certain operations. Uploading the request result to the temporary directory prevents other devices from reading incomplete request results from the shared disk when the target device is uploading the request result. This avoids unstable data reads when distributed applications perform simultaneous read and write operations. This is because, with this method, a soft link needs to be established before each connected device (i.e., the device connected to the shared storage) can read the request result based on the soft link, thus ensuring that the complete data is read. A soft link, also called a symbolic link, is similar to creating a shortcut for a file in Windows (a computer operating system). It creates a special file that points to another file. In other words, the soft link points to a requested result in a temporary directory, and various connected devices can access that requested result by reading the soft link. Essentially, the storage address of the soft link in shared memory represents the storage address of the requested result in shared memory. When a request result is associated with a request identifier and added to shared memory, the soft link can be associated with a corresponding request identifier. This identifier can then be used to determine the soft link containing the requested result, and the result can be retrieved. However, the actual requested result remains in the temporary directory of shared memory.

[0063] In one possible implementation, the request results can be uploaded to the temporary directory of the shared storage asynchronously. For example, rsync (a data mirroring and backup tool) can be used. Since rsync supports resuming interrupted uploads, it makes uploading the request results to the shared disk efficient and has a higher success rate. It is understood that, due to the asynchronous upload method, although shared storage is characterized by slow write operations, asynchronously uploading the request results to the shared storage does not affect other operations on the target device, thus not affecting data processing efficiency.

[0064] This application proposes a data processing scheme for shared memory, applicable to any one of multiple devices connected to the shared memory. Upon receiving a data processing request from a computing task for a distributed application, the scheme first checks the request result in local memory. If no corresponding request result is found in local memory, it checks the request result in shared memory. If no corresponding request result is found in shared memory, the request result is calculated in local memory and then added to the shared memory. Since any device connected to the shared memory calculates and uploads its request result to the shared memory, subsequent data processing requests can be processed by sequentially checking both local and shared memory for request results. This reduces performance overhead caused by redundant operations on multiple devices connected to the shared memory, thereby improving the processing efficiency of distributed application computing tasks.

[0065] Please see Figure 4 , Figure 4 This is a schematic flowchart illustrating a data processing method for shared memory provided in an embodiment of this application. The method can be executed by the aforementioned electronic device, which includes local memory and shared memory. The data processing method for shared memory may include the following steps.

[0066] S401. Receive a data processing request initiated by a computing task based on a distributed application, and detect the request result of the data processing request in the local storage.

[0067] S402. If the request result of the data processing request is not detected in the local memory, then the request result of the data processing request is detected in the shared memory.

[0068] Steps S401-S402 can be referred to the relevant descriptions of steps S301-S302, and will not be repeated here.

[0069] S403. If the request result of the data processing request is not detected in the shared memory, then the target file indicated by the data processing request is obtained.

[0070] The target file is used to indicate the files required by the data processing request during the processing. For example, if a data processing request instructs the user to obtain the coverage of compiled file A and executable file B, then the target files indicated by the data processing request are compiled file A and executable file B. It can be understood that obtaining the target file indicated by the data processing request here means writing the target file to local memory. Since shared memory has slow write operations, writing the target file to local memory is faster than writing the target file to shared memory and then reading it from shared memory for data processing, thus making the entire data processing process more efficient.

[0071] In one possible implementation, acquiring the target file indicated by the data processing request specifically includes the following steps: ① Detecting the target file indicated by the data processing request in local storage. It is understood that if the target file has been acquired in response to a data processing request requiring the acquisition of the target file before acquiring the target file indicated by the data processing request, the target file may be stored in the local storage of the target device. In one possible implementation, the data processing request carries a file identifier of the target file, and each file in the local storage has a corresponding file identifier. Therefore, the file identifier of the target file can be matched with the file identifiers of each file in the local storage to detect whether the target file exists in the local storage. If a file matching the file identifier of the target file exists in the local storage, it is determined that the target file has been detected in the local storage; otherwise, it is determined that the target file has not been detected in the local storage.

[0072] ② If the target file is detected in local storage, it is extracted from local storage. In one possible implementation, extracting the target file from local storage can be done by placing it in the file directory corresponding to the data processing request. This file directory can be used to store files involved in the processing of the data processing request. In one possible implementation, the target file detected in local storage is typically located in a temporary directory of local storage. Therefore, it can be extracted to the file directory corresponding to the data processing request by creating a symbolic link. Specifically, a symbolic link can be created in the file directory corresponding to the data processing request, pointing to the target file stored in the temporary directory of local storage.

[0073] ③ If the target file is not detected in the local storage, it is retrieved from the cloud storage system. As mentioned above, this cloud storage system can be used to store source files required for data processing requests, such as the target file in this step. Retrieving the target file from the cloud storage system means transferring the target file from the cloud storage system to the local storage. In one possible implementation, retrieving the target file from the cloud storage system can be done based on the target file's file identifier.

[0074] In one possible implementation, when any device connected to the shared storage processes a data processing request, it can add the target file indicated by the data processing request to the shared storage. If the target file is not detected in the local storage, it can be detected in the shared storage. If the target file is not detected in the shared storage, it is retrieved from the cloud storage system to the local storage; if the target file is detected in the shared storage, it is extracted from the shared storage to the local storage. Extracting the target file from the shared storage to the local storage can be done by: pulling the target file from the shared storage to a temporary directory of the local storage; and creating a symbolic link corresponding to the request result, pointing to the target file in the temporary directory of the local storage. Creating the symbolic link corresponding to the target file can be done in the file directory corresponding to the data processing request, so that when performing data processing based on the target file, the target file is read based on the symbolic link.

[0075] In one possible implementation, retrieving a target file from a cloud storage system includes: ① storing the target file retrieved from the cloud storage system to local storage. This retrieval can be done using a soft link, specifically including the following steps: retrieving the target file to a temporary directory on the local storage system; and creating a soft link corresponding to the target file, pointing to the target file in the temporary directory. The relevant description of soft links is as described above and will not be repeated here. Creating the soft link can be done in the file directory corresponding to the data processing request, so that when performing data processing based on the target file, the target file is read based on this soft link. In one possible implementation, the target file retrieved from the cloud storage system is typically a compressed file. After retrieving the compressed target file from the cloud storage system, the compressed target file can be decompressed to obtain the target file, stored in a temporary directory, and a soft link pointing to the decompressed target file in the temporary directory can be created. Therefore, when performing data processing based on the target file, the target file is read based on this soft link.

[0076] ② Generate the storage address of the target file in local storage. This storage address refers to the address required for the target device to read the target file from local storage. In other words, the storage address of the target file in local storage is the same as the storage address of the symbolic link corresponding to the target file, not the storage address of the temporary directory of the target file in local storage. The target device can then read the target file from local storage based on this address and calculate the requested result based on the read target file in local storage.

[0077] In one possible implementation, any one of the multiple devices connected to the shared memory contains multiple data acquisition threads. The target file is then retrieved from the cloud storage system. Specifically, this may include the following steps: ① If a target data acquisition thread is detected executing the step of retrieving the target file from the cloud storage system, the target file is locked. Here, a thread refers to an execution path within a program, and this data acquisition thread is used to indicate the thread that needs to retrieve data. If the target device receives many concurrent requests, the target device may contain multiple data acquisition threads. This target data acquisition thread is the one that executes the step of retrieving the target file from the cloud storage system first among the multiple data acquisition threads. Locking the target file means preventing other data acquisition threads besides this target data acquisition thread from writing the target file to local storage; that is, the locked target file cannot be retrieved by any data acquisition thread other than the target data acquisition thread. This ensures that only one data acquisition thread in the target device can retrieve the target file from the cloud storage system at any given time, preventing multi-threaded concurrent operations.

[0078] ② If the target data acquisition thread successfully retrieves the target file from the cloud storage system, the lock on the target file is released. Releasing the lock on the target file means that the target file can be retrieved by any of the multiple data acquisition threads other than the target data acquisition thread. As mentioned above, the successful retrieval of the target file from the cloud storage system by the target data acquisition thread may include storing the target file retrieved from the cloud storage system to local storage and generating the storage address of the target file in local storage; this will not be elaborated here. In one possible implementation, if the target data acquisition thread successfully retrieves the target file, other target data acquisition threads that need to retrieve the target file do not need to retrieve it from the cloud storage system again and can directly use the target file. This prevents duplicate retrieval of the same target file, i.e., prevents repeated write operations on the target file, reducing resource waste. In another possible implementation, if the process of retrieving the target file from the cloud storage system terminates abnormally, the lock on the target file also needs to be released to avoid deadlock, which would prevent other data acquisition threads from retrieving the target file normally.

[0079] S404. Calculate the request result of the data processing request in the local storage based on the obtained target file, and add the calculated request result to the shared storage.

[0080] As described above, the application corresponding to the data processing request can be invoked to calculate the corresponding request result in local storage based on the obtained target file. Adding the calculated request result to shared storage can be referred to the relevant description in step S303, which will not be repeated here.

[0081] In one possible implementation, as described above, upon obtaining the target file, its storage address in local memory can be generated. Then, calculating the request result corresponding to the data processing request based on the obtained target file in local memory can specifically involve: reading the target file from local memory according to the storage address, and calculating the request result of the data processing request based on the read target file in local memory. It is understood that since this storage address is the storage address of the corresponding symbolic link, when reading the target file, it is not necessary to copy the target file to the file directory of the corresponding data processing request. Instead, the target file is read through the symbolic link into a temporary directory stored in local memory, and then the calculation is performed based on the target file read through the symbolic link to obtain the request result.

[0082] In one possible implementation, when adding the calculated request result to the shared memory, the calculated request result can first be stored in the target device's local memory, and then added to the shared memory from the local memory. Step S303 can then include the following steps: ① Storing the calculated request result in the local memory and establishing a storage address for the calculated request result in the local memory. This storage address can be used to retrieve the request result later. It is understood that the request result is stored in the file directory corresponding to the data processing request in the local memory. ② Retrieving the request result from the local memory according to the storage address and adding the retrieved request result to the shared memory. This allows the calculated request result to be added from the local memory to the shared memory, so that other devices can retrieve the calculated request result from the shared memory.

[0083] In one possible implementation, as described above, the request result can be added to the shared storage in the form of a soft link, which will not be elaborated here.

[0084] In one possible implementation, since adding data to shared storage takes time, multiple devices connected to the shared disk may execute data processing requests that require calculating the same result within a short period. This means multiple devices may obtain the same result. However, when uploading the result to shared storage, it's necessary to prevent multiple devices from simultaneously uploading the same result. This can be achieved by the following steps: ① If the result is detected as unlocked in shared storage, the step of adding the result to shared storage is executed, and the target result is locked. Locking the result ensures that other devices connected to the shared device write the same result to shared storage while the result is being uploaded. Detecting that the target result is unlocked indicates that it is not being added to shared storage by other devices. If the result is detected as locked in shared storage, the step of adding it is not executed. This is because if the result is locked, it means another device is uploading it, and the device does not need to upload it again, preventing duplicate uploads.

[0085] ② If it is detected that the target request result has been successfully added to the shared memory, then the lock on the target request result is released. Releasing the lock on the target request result means that the target request result can be uploaded to the shared memory by other connected devices besides this device. In one possible implementation, if the request result is successfully added to the shared memory, other connected devices no longer need to upload the request result of the data processing request, thereby preventing the repeated uploading of multiple identical request results, i.e., preventing repeated write operations on the request result in the shared memory, and reducing resource waste. In another possible implementation, if the process of adding the request result to the shared memory terminates abnormally, it is also necessary to release the lock on the request result to avoid deadlock, which would prevent other connected devices from uploading the request result normally.

[0086] In one possible implementation, after executing the data retrieval request for the target file, the target file can also be added to shared storage. The method for adding the target file to shared storage can be similar to the method described above for adding the request result to the target file. Specifically, the target file can be retrieved based on its storage address in local storage, and then the target file retrieved from local storage can be added to shared storage.

[0087] When adding a target file retrieved from local storage to shared storage, if the target file is detected to be unlocked in shared storage, the process of adding the target file to shared storage is executed, and the target file is then locked. If the target file is successfully added to shared storage, the lock on the target file is released. This prevents other connected devices from uploading the target file to shared storage, avoids repeated write operations on the target file in shared storage, and reduces resource waste.

[0088] When adding a target file to shared storage, it can be done using a soft link. This involves uploading the target file to a temporary directory on the shared storage, and then creating a soft link in the shared storage pointing to the target file in the temporary directory. Essentially, creating a soft link in the shared storage corresponds to creating a soft link in the directory of the file associated with the data processing request. This ensures that both the target file indicated by the data processing request and the request result are stored in the shared storage. Devices connected to the shared storage can then retrieve the target file from there, reducing the need to repeatedly retrieve the target file from cloud storage, improving the efficiency of target file retrieval, and consequently, improving the processing efficiency of the computing task.

[0089] In one possible implementation, since both the target file and the computational request results can be added to the shared storage, the data in the local storage can be cleaned up periodically, reducing the storage space occupied in the local storage.

[0090] This section uses an example to illustrate the entire data processing procedure; please refer to [link / reference]. Figure 5 , Figure 5 This is a schematic diagram illustrating the framework of a data processing method for shared memory provided in an embodiment of this application. For example... Figure 5 As shown, this data processing system can include multiple data processing devices. First, the `mount` command can be used to mount shared storage to the same file directory on each of the distributed data processing devices, making each data processing device a mounted device for the shared storage. This allows each data processing device to include both local and shared storage. Here, mounted device 1 is used as the target device to describe the data processing process.

[0091] The data processing request is routed to the connected device 1. That is, after receiving the data processing request, connected device 1 determines the corresponding application to process it. First, it can check if the request result (e.g., data processing result) exists in the local storage. Figure 5 As shown in step ①, if the requested result does not exist in local memory, then check if the requested result exists in shared memory (e.g., ...). Figure 5 As shown in step ②), if the requested result is not present in the shared storage, the target file can be retrieved from the cloud storage system (e.g., ...). Figure 5 As shown in 501), specifically, a file lock can be established in local storage for the target file (such as...). Figure 5 As shown in 502), the target file is written to a temporary directory on the local storage (e.g., ...). Figure 5 As shown in 503), a soft link is then created pointing to the target file in the temporary directory of the local storage (e.g., ...). Figure 5 (As shown in 504). Furthermore, the target file can be added from local storage to shared storage (e.g., ...). Figure 5 As shown in step ③), specifically, a file lock for the target file can be established in the shared memory (e.g., ...). Figure 5 As shown in 505), the target file is written to a temporary directory of shared storage (e.g., ...). Figure 5 As shown in 506), a soft link is then created pointing to the target file in the temporary directory of the shared memory (e.g., ...). Figure 5 (As shown in 507). Furthermore, the connected device 1 can call an application in its local storage to perform calculations based on the obtained target file and obtain the requested result (such as...). Figure 5As shown in 508), and add the request result to the shared memory (e.g., Figure 5 (See step ④ in the text). Specifically, adding the request result to shared storage can involve establishing a file lock on the request result in the shared storage, writing the request result to a temporary directory in the shared storage, and then creating a symbolic link pointing to the request result in the temporary directory in the shared storage (e.g., ...). Figure 5 As shown in Figure 509, when uploading the request result and the target file to shared storage, file locks are used and the files are stored in a temporary directory. However, in practice, the file locks used apply to different files, which are not distinguished in this illustration and are only provided as an example. This allows us to implement the entire processing process for data processing requests.

[0092] In one possible implementation, a flowchart illustrating the method for obtaining request results is provided below. Please refer to [link to relevant documentation]. Figure 6 , Figure 6 This is a flowchart illustrating a method for obtaining request results provided in an embodiment of this application. Figure 6 As shown, the target device receives a data processing request (such as...). Figure 6 As shown in 601), the target device then detects whether there is a request result for the data processing request in the local memory (such as...). Figure 6 As shown in 602), if the request result of the data processing request exists in the local memory, then the request result is retrieved from the local memory (e.g., ...). Figure 6 (As shown in 603); if the request result for the data processing request does not exist in the local memory, then check if the request result for the data processing request exists in the shared memory (e.g., Figure 6 (As shown in 604). If the request result for the data processing request exists in the shared memory, then the request result is retrieved from the shared memory (e.g., ...). Figure 6 (As shown in 605); if the request result for the data processing request does not exist in the shared storage, then the target file (such as...) is retrieved from the cloud storage system. Figure 6 As shown in 606), and then in the local memory, the request result corresponding to the data processing request is calculated based on the obtained target file (such as...). Figure 6 As shown in 607), after obtaining the request result, the computing tasks of distributed applications (such as...) can be processed based on the request result. Figure 6 As shown in 608, the previously calculated request results can be retrieved from the local memory first. If the request result is not in the local memory, the previously calculated request results of each connected device can be retrieved from the shared memory. This avoids the repeated calculation of the same request result in each connected device, which helps to save computing resources and greatly improves the efficiency of obtaining request results, thereby improving the processing efficiency of computing tasks.

[0093] This application proposes a data processing scheme for shared memory, applicable to any one of multiple devices connected to the shared memory. Upon receiving a data processing request from a computing task for a distributed application, the scheme first checks the request result in local memory. If no corresponding request result is found in local memory, it checks the request result in shared memory. If no corresponding request result is found in shared memory, the request result is calculated in local memory and then added to the shared memory. Since any device connected to the shared memory calculates and uploads its request result to the shared memory, subsequent data processing requests can be processed by sequentially checking both local and shared memory for request results. This reduces performance overhead caused by redundant operations on multiple devices connected to the shared memory, thereby improving the processing efficiency of distributed application computing tasks.

[0094] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a data processing device for a shared memory according to an embodiment of this application. Optionally, this data processing device for the shared memory can be disposed in the aforementioned electronic device. Figure 7 As shown, the data processing device for the shared memory described in this embodiment may include:

[0095] The receiving unit 701 is used to receive data processing requests initiated by computing tasks based on distributed applications and to detect the request results of the data processing requests in local memory; multiple devices are used to process computing tasks of distributed applications.

[0096] The processing unit 702 is configured to detect the request result of the data processing request in the shared memory if the request result of the data processing request is not detected in the local memory.

[0097] The processing unit 702 is further configured to, if no request result for the data processing request is detected in the shared memory, calculate the request result for the data processing request in the local memory and add the calculated request result to the shared memory.

[0098] In one implementation, the processing unit 702 is specifically used for:

[0099] A request identifier for generating data processing requests;

[0100] The request result corresponding to the data processing request is detected in the shared memory based on the request identifier.

[0101] In one implementation, the processing unit 702 is specifically used for:

[0102] The request identifier is associated with the computed request result and added to the shared memory.

[0103] In one implementation, the processing unit 702 is further configured to:

[0104] If the result of the data processing request is detected in local memory, then the result of the data processing request is retrieved from local memory.

[0105] The computational tasks of distributed applications are processed based on the request results retrieved from local storage.

[0106] Processing unit 702 is also used for:

[0107] If the request result of the data processing request is detected in the shared memory, then the request result of the data processing request is retrieved from the shared memory.

[0108] The computational tasks of distributed applications are processed based on the request results retrieved from shared memory.

[0109] In one implementation, the processing unit 702 is specifically used for:

[0110] If no result of the data processing request is detected in the shared memory, then the target file indicated by the data processing request is obtained;

[0111] The request result of the data processing request is calculated in local storage based on the obtained target file.

[0112] In one implementation, the processing unit 702 is specifically used for:

[0113] Detect the target file indicated by the data processing request in local storage;

[0114] If the target file is detected in local storage, then extract the target file from local storage;

[0115] If the target file is not detected in the local storage, it will be retrieved from the cloud storage system.

[0116] In one implementation, the processing unit 702 is specifically used for:

[0117] The target files retrieved from the cloud storage system will be stored on the local storage device;

[0118] Generate the storage address of the target file in local storage;

[0119] The target file is read from the local storage based on the storage address, and the request result of the data processing request is calculated in the local storage based on the read target file.

[0120] In one implementation, any one of the multiple devices includes multiple data acquisition threads; the processing unit 702 is specifically used for:

[0121] If it is detected that the target data acquisition thread among multiple data acquisition threads is performing the step of pulling the target file from the cloud storage system, then the target file is locked;

[0122] If the target data acquisition thread successfully retrieves the target file from the cloud storage system, the lock on the target file will be released.

[0123] Specifically, the locked target file cannot be acquired by any data acquisition thread other than the target data acquisition thread among the multiple data acquisition threads.

[0124] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device described in this embodiment includes: a processor 801 and a memory 802. Optionally, the electronic device may also include a network interface or a power supply module, etc. The processor 801 and the memory 802 can exchange data. The memory 802 may include local memory and shared memory.

[0125] The processor 801 described above can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0126] The aforementioned network interface may include input devices and / or output devices. For example, the input device may be a control panel, microphone, receiver, etc., and the output device may be a display screen, transmitter, etc., which are not listed here. For instance, in the embodiments of the application, the network interface may include a receiver and a transmitter.

[0127] The aforementioned memory 802 may include read-only memory and random access memory, and provides program instructions and data to the processor 801. A portion of memory 802 may also include non-volatile random access memory. The processor 801 executes program instructions when it calls them.

[0128] It receives data processing requests initiated by computing tasks based on distributed applications and detects the request results in local storage; multiple devices are used to process computing tasks of distributed applications.

[0129] If the result of the data processing request is not detected in local memory, the result of the data processing request is detected in shared memory.

[0130] If no result for the data processing request is detected in the shared memory, the result for the data processing request is calculated in the local memory and then added to the shared memory.

[0131] In one implementation, processor 801 is specifically used for:

[0132] A request identifier for generating data processing requests;

[0133] The request result corresponding to the data processing request is detected in the shared memory based on the request identifier.

[0134] In one implementation, processor 801 is specifically used for:

[0135] The request identifier is associated with the computed request result and added to the shared memory.

[0136] In one implementation, the processor 801 is also used for:

[0137] If the result of the data processing request is detected in local memory, then the result of the data processing request is retrieved from local memory.

[0138] The computational tasks of distributed applications are processed based on the request results retrieved from local storage.

[0139] Processor 801 is also used for:

[0140] If the request result of the data processing request is detected in the shared memory, then the request result of the data processing request is retrieved from the shared memory.

[0141] The computational tasks of distributed applications are processed based on the request results retrieved from shared memory.

[0142] In one implementation, processor 801 is specifically used for:

[0143] If no result of the data processing request is detected in the shared memory, then the target file indicated by the data processing request is obtained;

[0144] The request result of the data processing request is calculated in local storage based on the obtained target file.

[0145] In one implementation, processor 801 is specifically used for:

[0146] Detect the target file indicated by the data processing request in local storage;

[0147] If the target file is detected in local storage, then extract the target file from local storage;

[0148] If the target file is not detected in the local storage, it will be retrieved from the cloud storage system.

[0149] In one implementation, processor 801 is specifically used for:

[0150] The target files retrieved from the cloud storage system will be stored on the local storage device;

[0151] Generate the storage address of the target file in local storage;

[0152] The target file is read from the local storage based on the storage address, and the request result of the data processing request is calculated in the local storage based on the read target file.

[0153] In one implementation, any one of the multiple devices includes multiple data acquisition threads; processor 801 is specifically used for:

[0154] If it is detected that the target data acquisition thread among multiple data acquisition threads is performing the step of pulling the target file from the cloud storage system, then the target file is locked;

[0155] If the target data acquisition thread successfully retrieves the target file from the cloud storage system, the lock on the target file will be released.

[0156] Specifically, the locked target file cannot be acquired by any data acquisition thread other than the target data acquisition thread among the multiple data acquisition threads.

[0157] Optionally, when the program instructions are executed by the processor, other steps of the method in the above embodiments can also be implemented, which will not be described in detail here.

[0158] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the methods described above, such as the methods performed by the electronic device described above, which will not be elaborated here.

[0159] Optionally, the storage medium involved in this application, such as a computer-readable storage medium, may be non-volatile or volatile.

[0160] Optionally, the computer-readable storage medium may primarily include a stored program area and a stored data area. The stored program area may store the operating system, at least one application program required for a given function, etc.; the stored data area may store data created based on the use of blockchain nodes, etc. Here, the blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and cryptographic algorithms. A blockchain is essentially a decentralized database, a chain of data blocks linked using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain may include a blockchain underlying platform, a platform product service layer, and an application service layer, etc.

[0161] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0162] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0163] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps described in the embodiments of the methods above. For example, the computer device may be a terminal or a server.

[0164] The data processing method, related devices, products, and media of the shared memory provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data processing method for a shared memory, characterized in that, The shared memory is connected to multiple devices, and the method is applied to any one of the multiple devices. The method includes: The system receives data processing requests initiated by computing tasks based on distributed applications and detects the request results of the data processing requests in local storage; the multiple devices are used to process the computing tasks of the distributed applications. If the request result of the data processing request is not detected in the local memory, then the request result of the data processing request is detected in the shared memory; If the request result of the data processing request is not detected in the shared memory, the request result of the data processing request is calculated in the local memory and added to the shared memory.

2. The method according to claim 1, characterized in that, The step of detecting the request result of the data processing request in the shared memory includes: Generate a request identifier for the data processing request; The request result corresponding to the data processing request is detected in the shared memory based on the request identifier.

3. The method according to claim 2, characterized in that, Adding the calculated request result to the shared memory includes: The request identifier is associated with the calculated request result and added to the shared memory.

4. The method according to claim 1, characterized in that, The method further includes: If the request result of the data processing request is detected in the local memory, then the request result of the data processing request is extracted from the local memory. The distributed application's computational tasks are processed based on the request results extracted from the local storage. The method further includes: If the request result of the data processing request is detected in the shared memory, then the request result of the data processing request is extracted from the shared memory; The distributed application's computational tasks are processed based on the request results extracted from the shared memory.

5. The method according to claim 1, characterized in that, If the request result of the data processing request is not detected in the shared memory, the request result of the data processing request is calculated in the local memory, including: If the request result of the data processing request is not detected in the shared memory, then the target file indicated by the data processing request is obtained; The request result of the data processing request is calculated in the local storage based on the obtained target file.

6. The method according to claim 5, characterized in that, The step of obtaining the target file indicated by the data processing request includes: The target file indicated by the data processing request is detected in the local storage; If the target file is detected in the local storage, then the target file is extracted from the local storage; If the target file is not detected in the local storage, the target file is retrieved from the cloud storage system.

7. The method according to claim 6, characterized in that, The step of retrieving the target file from the cloud storage system includes: The target file retrieved from the cloud storage system is stored in the local storage device; Generate the storage address of the target file in the local memory; The step of calculating the request result of the data processing request in the local storage based on the obtained target file includes: The target file is read from the local storage according to the storage address, and the request result of the data processing request is calculated in the local storage based on the read target file.

8. The method according to claim 6, characterized in that, Each of the plurality of devices contains a plurality of data acquisition threads; The step of retrieving the target file from the cloud storage system includes: If it is detected that the target data acquisition thread among the plurality of data acquisition threads is performing the step of pulling the target file from the cloud storage system, then the target file is locked; If it is detected that the target data acquisition thread has successfully retrieved the target file from the cloud storage system, then the lock on the target file is released; Specifically, the locked target file is not acquired by any data acquisition thread other than the target data acquisition thread among the plurality of data acquisition threads.

9. A data processing apparatus for a shared memory, characterized in that, The shared memory is connected to multiple devices, and the device is disposed in any one of the multiple devices. The device includes: A receiving unit is configured to receive data processing requests initiated by computing tasks based on distributed applications, and detect the request results of the data processing requests in local memory; the plurality of devices are configured to process the computing tasks of the distributed applications; A processing unit is configured to detect the request result of the data processing request in the shared memory if the request result of the data processing request is not detected in the local memory. The processing unit is further configured to, if no request result of the data processing request is detected in the shared memory, calculate the request result of the data processing request in the local memory and add the calculated request result to the shared memory.

10. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to invoke the program instructions to perform the method as claimed in any one of claims 1-8.

11. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the method of any one of claims 1-8.

12. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method as claimed in any one of claims 1-8.

Images