Data processing method and device based on diffusion model, equipment and storage medium

Abstract

The invention can be applied to the field of intelligent medical treatment and finance, and discloses a data processing method, device and equipment based on a diffusion model, and a storage medium, and the method comprises the steps: obtaining data to be inputted to a pre-trained diffusion model for processing, and carrying out the preprocessing of the obtained data; carrying out distributed storage on the preprocessed data; dividing an iterative computation task of the diffusion model into a plurality of computation sub-tasks by utilizing a MapReduce framework, and computing each computation sub-task in parallel according to each data block stored in a distributed manner; and fusing the calculation results of the calculation subtasks obtained by parallel calculation to obtain a diffusion model calculation result. Through big data distributed storage and a MapReduce framework-based parallel computing technology, a computing task of the diffusion model can be decomposed and executed in parallel on a plurality of computing nodes, so that the data processing efficiency of the diffusion model in a big data scene is remarkably improved, and a data analysis scene with a relatively high real-time requirement can be met.

Description

Technical Field

[0001] The present invention relates to the field of big data processing technology, and in particular to a data processing method, device, equipment and storage medium based on a diffusion model. Background Art

[0002] Diffusion models are well-known for their ability to generate high-quality data (such as images, videos, and audio) by gradually adding and reversing noise. They have demonstrated powerful capabilities in a wide range of fields, including data analysis, image generation, and signal processing. In recent years, diffusion models have demonstrated their practicality in areas such as medical image analysis (such as lesion detection) and financial business scenarios (such as transaction target detection). Their high-fidelity output has become crucial for the technology's implementation.

[0003] However, with the advent of the big data era, the amount of data in medical consultation scenarios and financial transaction businesses has grown exponentially. Because the Diffusion model requires a large number of iterative calculations when calculating the evolution of probability distribution during the diffusion process, this makes the Diffusion model extremely time-consuming when processing large-scale data. It cannot efficiently meet the needs of processing and analyzing massive data, which seriously limits its application in scenarios with high real-time requirements. Summary of the Invention

[0004] The present invention provides a data processing method, device, equipment and storage medium based on a diffusion model to solve the technical problem that the existing diffusion model takes a very long time to process large-scale data and cannot efficiently meet the processing and analysis needs of massive data.

[0005] In a first aspect, a data processing method based on a diffusion model is provided, comprising:

[0006] Acquire data to be input into the pre-trained diffusion model for processing, and pre-process the acquired data;

[0007] Distribute and store the pre-processed data;

[0008] The MapReduce framework is used to divide the iterative computing task of the diffusion model into multiple computing subtasks, and each computing subtask is calculated in parallel based on the distributed storage data blocks;

[0009] The calculation results of each subtask obtained by parallel calculation are integrated to obtain the diffusion model calculation results.

[0010] In a second aspect, a data processing device based on a diffusion model is provided, comprising:

[0011] A preprocessing module is used to obtain data to be input into the pre-trained diffusion model for processing and to preprocess the obtained data;

[0012] Storage module, used for distributed storage of pre-processed data;

[0013] The parallel computing module is used to divide the iterative computing task of the diffusion model into multiple computing subtasks using the MapReduce framework, and to parallelly compute each computing subtask based on the distributed storage data blocks;

[0014] The fusion optimization module is used to fuse the calculation results of each calculation subtask obtained by parallel calculation to obtain the diffusion model calculation result.

[0015] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned data processing method when executing the computer program.

[0016] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned data processing method are implemented.

[0017] In the scheme implemented by the above-mentioned data processing method, device, equipment and storage medium based on the diffusion model, data to be input into the pre-trained diffusion model for processing can be obtained through the client or from the server, and the obtained data can be preprocessed; the preprocessed data is then distributedly stored; and the iterative calculation task of the diffusion model is divided into multiple calculation subtasks using the MapReduce framework, and each calculation subtask is calculated in parallel according to each data block in the distributed storage; finally, the calculation results of each calculation subtask obtained by the parallel calculation are integrated to obtain the diffusion model calculation result. In the present invention, when data processing is performed based on the diffusion model in scenarios such as medical care, finance, and insurance, for example, when medical image analysis is performed based on the diffusion model during doctor-patient consultation or target detection is performed based on the diffusion model during financial transactions, the calculation tasks of the diffusion model can be decomposed and executed in parallel on multiple computing nodes through big data distributed storage and parallel computing technology based on the MapReduce framework, which greatly shortens the time for data processing and analysis, significantly improves processing efficiency, can meet data processing and analysis scenarios with high real-time requirements, solves the problem of slow processing speed of the diffusion model, and improves the efficiency of data processing of the diffusion model in big data scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 This is a schematic diagram of an application environment of a data processing method based on a diffusion model according to an embodiment of the present invention;

[0020] Figure 2 This is a flow chart of a data processing method based on a diffusion model in a first embodiment of the present invention;

[0021] Figure 3 yes Figure 2 A schematic flow chart of a specific implementation of step S130;

[0022] Figure 4 This is a flow chart of a data processing method based on a diffusion model in a second embodiment of the present invention;

[0023] Figure 5 is a structural diagram of a data processing device based on a diffusion model in one embodiment of the present invention;

[0024] Figure 6 is a structural diagram of a computer device in one embodiment of the present invention;

[0025] Figure 7 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0027] The data processing method based on the diffusion model provided by the embodiment of the present invention can be applied in the following fields: Figure 1In an application environment, a client communicates with a server via a network. The server can obtain data to be input into a pre-trained diffusion model for processing through the client or periodically from the network, preprocess the obtained data, and then distribute the preprocessed data. The MapReduce framework is used to divide the iterative calculation task of the diffusion model into multiple subtasks. Each subtask is then calculated in parallel based on the distributed stored data blocks. Finally, the calculation results of each subtask obtained by the parallel calculation are integrated to obtain the diffusion model calculation result. Specifically, the above-mentioned data processing method based on the diffusion model can be applied to intelligent information interaction platforms in scenarios such as medical care, finance, and insurance. For example, data processing can be performed based on the diffusion model during medical consultation (such as medical image analysis) or during financial transactions (such as transaction target detection). Through big data distributed storage and parallel computing technology based on the MapReduce framework, the computing tasks of the diffusion model can be decomposed and executed in parallel on multiple computing nodes, which greatly shortens the time for data processing and analysis, significantly improves processing efficiency, and can efficiently cope with the processing and analysis needs of massive data. It can meet data processing and analysis scenarios with high real-time requirements, solves the problem of slow processing speed of the diffusion model, and improves the efficiency of data processing of the diffusion model in big data scenarios. In the present invention, the client can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices, and the server can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0028] See also Figure 2 As shown, Figure 2 A flowchart of a data processing method based on a diffusion model provided in the first embodiment of the present invention includes the following steps S110-S140:

[0029] S110: Acquire data to be input into the pre-trained diffusion model for processing, and pre-process the acquired data.

[0030] In this step, the acquired data is cleaned to remove noise data, duplicate data and / or data with incorrect format, and feature extraction is performed on the cleaned data. In this embodiment, the data may be image, text, audio, or other data.

[0031] Understandably, when cleaning data, for numerical data such as audio, a pre-set value range can be used, and data outside the range can be treated as noise and removed. For text data, regular expression matching can be used to remove data that does not meet format requirements. When extracting features from the cleaned data, the corresponding feature extraction methods are also used for different types of data. For example, for image data, convolutional neural networks are used to extract the image's feature vectors. For structured numerical and textual data, highly correlated feature columns are selected as input features for analysis, and the extracted features can be standardized or normalized to facilitate subsequent calculations.

[0032] The data processing method based on the diffusion model provided by the present invention can be applied to intelligent information interaction platforms in various application scenarios such as the smart medical field and the financial technology field. It is usually implemented through a server, which can obtain the data to be input in real time. For example, in the field of smart medical care, when conducting online consultations, users often use intelligent information interaction platforms such as intelligent customer service or intelligent assistants to consult and ask questions through online chat. Medical personnel can use the diffusion model in the intelligent information interaction platform to perform data processing and analysis, thereby achieving rapid consultation and response. For example, when the user is conducting an online consultation, the data to be input is a nuclear magnetic resonance image of the lungs. To know whether there is a lesion in the nuclear magnetic resonance image of the lungs, the intelligent customer service can use the data processing method based on the diffusion model provided by the present invention on the server to perform lesion detection in the medical image, thereby quickly identifying and locking the target lesion, helping medical personnel to quickly analyze and respond; or, it can also be used to read medical diagnosis information, realizing the reading, storage and image generation of a large amount of medical diagnosis information. In the application scenarios of insurance, banking, securities and other businesses, for car insurance accident reporting services, users can report accidents through the online platform, and car insurance business personnel can also use the diffusion model in the intelligent information interaction platform to detect the damage location and damage degree, thereby quickly identifying and locking the vehicle damage location and knowing the corresponding damage degree.

[0033] S120: Distributed storage of the pre-processed data.

[0034] In this step, the preprocessed data is divided into multiple data blocks using the Hadoop Distributed File System (HDFS). Each data block is replicated to obtain multiple copies of each data block. The data blocks and copies are then distributed and stored across different storage nodes in the cluster to improve data reliability and access speed. As will be appreciated, after the multiple data blocks are distributed, a data block index table records the storage location and related attribute information of each data block, including the data type (e.g., image data or text data).

[0035] Preferably, in this embodiment, there are three copies of each data block, and the data blocks and copies are distributed and stored on different storage nodes of the distributed file system. At least one of the three copies can be stored on a different rack through a rack-aware strategy.

[0036] For example, when using the diffusion model-based data processing method provided by the present invention to detect lung lesions in magnetic resonance imaging (MRI) images, the MRI image can first be divided into multiple data blocks based on coordinate location. Each data block can be replicated multiple times to generate three copies. The image data blocks and copies are then distributed and stored, with one copy and the remaining two copies stored on different nodes in different racks. In the present invention, the distributed storage method ensures data reliability through a data block replication mechanism. Even if some nodes fail, data will not be lost, ensuring the stable operation of the data processing method.

[0037] S130: using the MapReduce framework to divide the iterative calculation task of the diffusion model into multiple calculation subtasks, and calculating each calculation subtask in parallel based on the distributed stored data blocks.

[0038] Specifically, if Figure 3 As shown, step 130 includes the following steps S1301-S1302:

[0039] S1301: Divide the iterative calculation task of the diffusion model into multiple calculation subtasks.

[0040] In the present invention, the diffusion model usually involves multiple iterations. For example, data is gradually generated during the denoising process. When dividing the computing subtasks, each iterative calculation can be divided into one computing subtask, or each iterative computing task can be divided into multiple computing subtasks. For example, the denoising or diffusion process in each iterative calculation can be decomposed into multiple computing subtasks according to the time step. Each computing subtask processes a different data block, and computing subtasks of different time steps can be assigned to different computing nodes.

[0041] S1302: Each computing subtask is assigned to a computing node as a Map operation. Multiple computing nodes perform local diffusion calculations in parallel based on the distributed storage data blocks corresponding to the assigned computing subtasks, and merge the calculation results in a Reduce operation for analysis to obtain the diffusion calculation results of each computing subtask.

[0042] In this embodiment, each computational subtask can be assigned to a compute node as a Map operation according to the time step. Each compute node is responsible for performing a portion of the Diffusion model's computational tasks on the data block corresponding to the computational subtask. In the present invention, each computational subtask is converted into a MapReduce task upon execution. Each Map operation processes a data block, and the compute node performs a diffusion operation on the data block assigned to the current computational subtask. Specifically, in the Map phase, a forward diffusion process is performed to add noise to the data block, calculate the noise prediction for the current time step, and generate data for the next time step. In the Reduce phase, the local results are integrated to obtain the diffusion calculation results for each computational subtask.

[0043] For example, when performing lesion detection on distributedly stored MRI images, each computing node reads the data block assigned to its computing subtask from HDFS. In the Map phase, a preliminary Diffusion calculation is performed on the image features in the data block. In the Reduce phase, the local calculation results of each Map phase can be summarized to determine whether there is a lesion in the data block and the specific location of the lesion.

[0044] In the present invention, the data between each computing node can be synchronously managed and updated through the parameter server, that is, the result parameters of the computing subtasks at different time steps can be stored in the parameter server. When the computing node executes the computing task, it obtains the latest relevant parameters from the parameter server, and updates the result parameters to the parameter server after completing the calculation.

[0045] S140: Fusion the calculation results of each calculation subtask obtained by parallel calculation to obtain a diffusion model calculation result.

[0046] In this step, based on the index information of the data block, the calculation results of each calculation subtask are integrated according to the original data order or position order of the data to be input, and the integrated data is post-processed to obtain the diffusion model calculation results. For example, for the calculation results of image generation, the image enhancement algorithm is used for post-processing to improve the clarity and quality of the image, and for the data analysis results, statistical methods are used for post-processing to correct errors and verify the results.

[0047] It can be seen that in the above solution, when data processing is performed based on the diffusion model in the intelligent information interaction platform for scenarios such as smart medical care and financial technology, the computing tasks of the diffusion model can be decomposed and executed in parallel on multiple computing nodes through big data distributed storage and parallel computing technology based on the MapReduce framework, which greatly shortens the time for data processing and analysis, significantly improves processing efficiency, and can efficiently respond to the processing and analysis needs of massive data. It can meet the data processing and analysis scenarios with high real-time requirements, solves the problem of slow processing speed of the diffusion model, and improves the efficiency of data processing of the diffusion model in big data scenarios.

[0048] See also Figure 4 As shown, Figure 4 A flow chart of a data processing method based on a diffusion model provided in the second embodiment of the present invention includes the following steps S210-S250:

[0049] S210: Acquire data to be input into the pre-trained diffusion model for processing, and pre-process the acquired data.

[0050] This step is the same as or similar to step S110 and will not be described again here.

[0051] S220: Distributed storage of the pre-processed data.

[0052] This step is the same as or similar to step S120 and will not be described again here.

[0053] S230: using the MapReduce framework to divide the iterative computing task of the diffusion model into multiple computing subtasks, and computing each computing subtask in parallel based on the distributedly stored data blocks.

[0054] This step is the same as or similar to step S130 and will not be described again here.

[0055] S240: Fusion the calculation results of each calculation subtask obtained by parallel calculation to obtain a diffusion model calculation result.

[0056] This step is the same as or similar to step S140 and will not be described again here.

[0057] S250: Monitors the status of each computing node and storage node in HDFS in real time, and dynamically schedules computing subtasks and data storage distribution based on the status of each computing node and storage node to achieve load balancing.

[0058] In the present invention, all computing nodes and storage nodes periodically send heartbeat information and their current load status information to the master node (such as the server) to monitor the status of each computing node and storage node in HDFS in real time, so as to promptly discover node failures or performance bottlenecks.

[0059] Specifically, in this step, a dynamic weighted minimum load priority algorithm is used to dynamically schedule computing subtasks and data storage distribution according to the load status information of each computing node and each storage node, thereby achieving load balancing and ensuring efficient and stable operation of data processing. Preferably, in this embodiment, the load status information of the computing node includes the CPU usage, memory occupancy, and network bandwidth of the computing node, and the load status information of the storage node includes the current storage capacity and remaining storage space. For example, when it is found that the CPU usage of a computing node continues to exceed 80%, part of the computing tasks will be transferred from the computing node to other computing nodes with a CPU usage of less than 30% through the dynamic weighted minimum load priority algorithm; when it is found that the remaining storage space of a storage node is less than 10GB, the dynamic weighted minimum load priority algorithm can automatically store new data blocks on storage nodes with a remaining storage space greater than 50GB.

[0060] It can be seen that in the above scheme, when data processing is performed based on the diffusion model in the intelligent information interaction platform for scenarios such as smart medical care and financial technology, the data can be cleaned to improve the accuracy of data processing and analysis. During the processing, through big data distributed storage and parallel computing technology based on the MapReduce framework, the computing tasks of the diffusion model can be decomposed and executed in parallel on multiple computing nodes, which greatly shortens the time for data processing and analysis, significantly improves processing efficiency, and can meet data processing and analysis scenarios with high real-time requirements. It solves the problem of slow processing speed of the diffusion model and improves the data processing efficiency of the diffusion model in big data scenarios. At the same time, it can also monitor the status of the node and realize intelligent scheduling, so as to timely discover and handle node failures.

[0061] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0062] In one embodiment, a data processing device based on a diffusion model is provided, and the data processing device based on a diffusion model corresponds one-to-one to the data processing method based on a diffusion model in the second embodiment. Figure 5 As shown, the data processing device includes a pre-processing module 101, a storage module 102, a parallel computing module 103 and a fusion optimization module 104. The functional modules are described in detail as follows:

[0063] A preprocessing module 101 is used to obtain data to be input into the pre-trained diffusion model for processing and to preprocess the obtained data;

[0064] The storage module 102 is used for distributing and storing the pre-processed data;

[0065] The parallel computing module 103 is used to divide the iterative computing task of the diffusion model into multiple computing subtasks using the MapReduce framework, and to parallelly compute each computing subtask based on the distributed stored data blocks;

[0066] The fusion optimization module 104 is used to fuse the calculation results of each calculation subtask obtained by parallel calculation to obtain the diffusion model calculation result.

[0067] In one embodiment, the parallel computing module 103 is specifically configured to:

[0068] Divide the iterative calculation task of the diffusion model into multiple calculation subtasks;

[0069] Each computing subtask is assigned to a computing node as a Map operation. Multiple computing nodes perform local diffusion calculations in parallel based on the distributed storage data blocks corresponding to the assigned computing subtasks, and the calculation results are merged and analyzed in the Reduce operation to obtain the diffusion calculation results of each computing subtask.

[0070] In one embodiment, the storage module 102 is specifically configured to:

[0071] HDFS is used to split the preprocessed data into multiple data blocks, and each data block is replicated to obtain multiple copies of each data block. The data blocks and copies are then distributed and stored on different storage nodes in the cluster.

[0072] In one embodiment, the fusion optimization module 104 is further configured to:

[0073] Monitor the status of each computing node and storage node in HDFS in real time, and dynamically schedule computing subtasks and data storage distribution according to the status of each computing node and storage node to achieve load balancing.

[0074] In one embodiment, the fusion optimization module 104 is specifically configured to:

[0075] Using the dynamic weighted minimum load priority algorithm, computing subtasks and data storage distribution are dynamically scheduled according to the load status information of each computing node and each storage node to achieve load balancing.

[0076] In one embodiment, the fusion optimization module 104 is further configured to:

[0077] The calculation results of each calculation subtask are integrated according to the index information of the data block, and the integrated data is post-processed to obtain the diffusion model calculation results.

[0078] In one embodiment, the preprocessing module 101 is specifically configured to:

[0079] The acquired data is cleaned and features are extracted from the cleaned data.

[0080] The present invention provides a data processing device based on a diffusion model, which obtains data to be input into a pre-trained diffusion model for processing and pre-processes the obtained data; then distributes and stores the pre-processed data; and uses a MapReduce framework to divide the iterative calculation task of the diffusion model into multiple calculation subtasks, and parallelly calculates each calculation subtask according to each data block in the distributed storage; finally, the calculation results of each calculation subtask obtained by the parallel calculation are integrated to obtain the calculation result of the diffusion model, that is, during operation, the data can be cleaned to improve the accuracy of data processing and analysis. During the processing, the calculation tasks of the diffusion model can be decomposed and executed in parallel on multiple computing nodes through big data distributed storage and parallel computing technology based on the MapReduce framework, which greatly shortens the time of data processing and analysis, significantly improves processing efficiency, can meet data processing and analysis scenarios with high real-time requirements, solves the problem of slow processing speed of the diffusion model, and improves the data processing efficiency of the diffusion model in big data scenarios. At the same time, the node status can be monitored to realize intelligent scheduling, so as to timely discover and handle node failures.

[0081] The specific definition of the diffusion model-based data processing device can be found in the definition of the diffusion model-based data processing method described above and will not be further elaborated here. Each module in the aforementioned data processing device may be implemented in whole or in part via software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in the form of software in a memory in the computer device, so that the processor can call and execute the corresponding operations of each module.

[0082] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the server side of a data processing method based on a diffusion model.

[0083] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the client side of a data processing method based on a diffusion model.

[0084] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0085] Acquire data to be input into the pre-trained diffusion model for processing, and pre-process the acquired data;

[0086] Distribute and store the pre-processed data;

[0087] The MapReduce framework is used to divide the iterative computing task of the diffusion model into multiple computing subtasks, and each computing subtask is calculated in parallel based on the distributed storage data blocks;

[0088] The calculation results of each subtask obtained by parallel calculation are integrated to obtain the diffusion model calculation results.

[0089] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0090] Acquire data to be input into the pre-trained diffusion model for processing, and pre-process the acquired data;

[0091] Distribute and store the pre-processed data;

[0092] The MapReduce framework is used to divide the iterative computing task of the diffusion model into multiple computing subtasks, and each computing subtask is calculated in parallel based on the distributed storage data blocks;

[0093] The calculation results of each subtask obtained by parallel calculation are integrated to obtain the diffusion model calculation results.

[0094] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0095] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0096] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0097] The above embodiments are intended only to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they may still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein with equivalents. Such modifications or replacements do not deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention and are therefore intended to be included within the scope of protection of the present invention. Furthermore, any software tools or components not owned by the Company that appear in the embodiments of this application are merely examples and do not represent actual use.

Claims

1. A data processing method based on a diffusion model, characterized in that: include: Acquire data to be input into the pre-trained diffusion model for processing, and pre-process the acquired data; Distribute and store the pre-processed data; The MapReduce framework is used to divide the iterative computing task of the diffusion model into multiple computing subtasks, and each computing subtask is calculated in parallel based on the distributed storage data blocks; The calculation results of each subtask obtained by parallel calculation are integrated to obtain the diffusion model calculation results.

2. The data processing method based on the diffusion model according to claim 1, characterized in that: The MapReduce framework is used to divide the iterative calculation task of the diffusion model into multiple calculation subtasks, and each calculation subtask is calculated in parallel according to each data block of distributed storage, specifically including: Divide the iterative calculation task of the diffusion model into multiple calculation subtasks; Each computing subtask is assigned to a computing node as a Map operation. Multiple computing nodes perform local diffusion calculations in parallel based on the distributed storage data blocks corresponding to the assigned computing subtasks, and the calculation results are merged and analyzed in the Reduce operation to obtain the diffusion calculation results of each computing subtask.

3. The data processing method based on the diffusion model according to claim 1, characterized in that: The distributed storage of the pre-processed data specifically includes: HDFS is used to split the preprocessed data into multiple data blocks, and each data block is replicated to obtain multiple copies of each data block. The data blocks and copies are then distributed and stored on different storage nodes in the cluster.

4. The data processing method based on the diffusion model according to claim 3, characterized in that: The data processing method based on the diffusion model also includes: Monitor the status of each computing node and storage node in HDFS in real time, and dynamically schedule computing subtasks and data storage distribution according to the status of each computing node and storage node to achieve load balancing.

5. The data processing method based on the diffusion model according to claim 4, characterized in that: The dynamic scheduling of computing subtasks and data storage distribution based on the status of each computing node and storage node to achieve load balancing specifically includes: Using the dynamic weighted minimum load priority algorithm, computing subtasks and data storage distribution are dynamically scheduled according to the load status information of each computing node and each storage node to achieve load balancing.

6. The data processing method based on the diffusion model according to claim 1, characterized in that: The calculation results of each computing subtask obtained by parallel computing are integrated to obtain the diffusion model calculation result, which specifically includes: The calculation results of each calculation subtask are integrated according to the index information of the data block, and the integrated data is post-processed to obtain the diffusion model calculation results.

7. The data processing method based on the diffusion model according to claim 1, characterized in that: The preprocessing of the acquired data specifically includes: The acquired data is cleaned and features are extracted from the cleaned data.

8. A data processing device based on a diffusion model, characterized in that: include: A preprocessing module is used to obtain data to be input into the pre-trained diffusion model for processing and to preprocess the obtained data; Storage module, used for distributed storage of pre-processed data; The parallel computing module is used to divide the iterative computing task of the diffusion model into multiple computing subtasks using the MapReduce framework, and to parallelly compute each computing subtask based on the distributed storage data blocks; The fusion optimization module is used to fuse the calculation results of each calculation subtask obtained by parallel calculation to obtain the diffusion model calculation result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

Images