DICOM image preprocessing and quality control system based on Spark

Abstract

The invention relates to DICOM image processing, in particular to a DICOM image preprocessing and quality control system based on Spark, and the system comprises a data collection module which collects a DICOM medical image from a data source, and automatically recognizes and extracts a key metadata field in the DICOM medical image; the index construction module is used for constructing a comprehensive and accurate index system according to key metadata fields of the DICOM medical images; the data standardization module is used for carrying out standardization processing on key metadata fields of the DICOM medical images, supporting version control and audit tracking on the key metadata fields and carrying out optimization processing on the DICOM medical images; the scoring module is used for carrying out objective quality evaluation on the DICOM medical image based on a multi-dimensional image quality evaluation model and calculating a corresponding comprehensive quality score; according to the invention, the defect that the DICOM medical images from different equipment manufacturers are difficult to efficiently preprocess can be overcome.

Description

Technical Field

[0001] This invention relates to DICOM image processing, and more specifically to a Spark-based DICOM image preprocessing and quality control system. Background Technology

[0002] In modern healthcare, medical imaging data (such as X-rays, CT scans, and MRIs) has become an indispensable part of medical diagnosis and treatment planning. With advancements in medical technology, the number of DICOM (Digital Imaging and Communications in Medicine) format medical image files has exploded. These images are not only large in size but also contain a wealth of information, which is crucial for improving diagnostic accuracy and the quality of patient care.

[0003] However, current DICOM image preprocessing and standardization workflows face many challenges:

[0004] 1) Huge amount of data: Medical institutions generate an extremely large amount of DICOM medical image data every day, and traditional stand-alone processing methods are difficult to meet the needs of fast and efficient processing;

[0005] 2) High data heterogeneity: DICOM medical images from different equipment manufacturers may have inconsistent metadata formats and differences in image resolution, which increases the difficulty of unified management and analysis;

[0006] 3) Urgent need for standardization: In order to ensure image quality and consistency, a series of preprocessing operations must be performed on DICOM medical images, including denoising, enhancement, normalization and format conversion. However, existing preprocessing tools are often inefficient and cannot meet the needs of large-scale distributed environments.

[0007] 4) Strict compliance requirements: When processing sensitive medical data, it is imperative to strictly comply with relevant laws and regulations such as HIPAA (Health Insurance Portability and Accountability Act) to ensure data security and patient privacy protection. Summary of the Invention

[0008] (a) Technical problems to be solved

[0009] In view of the above-mentioned shortcomings of the existing technology, the present invention provides a Spark-based DICOM image preprocessing and quality control system, which can effectively overcome the shortcomings of the existing technology in that it is difficult to perform efficient preprocessing of DICOM medical images from different equipment manufacturers.

[0010] (II) Technical Solution

[0011] To achieve the above objectives, the present invention provides the following technical solution:

[0012] The Spark-based DICOM image preprocessing and quality control system includes a data acquisition module, an index building module, a data standardization module, a scoring module, a data storage module, and a processing workflow recording module.

[0013] The data acquisition module collects DICOM medical images from the data source and automatically identifies and extracts key metadata fields from the DICOM medical images;

[0014] The index building module constructs a comprehensive and accurate index system based on the key metadata fields of DICOM medical images;

[0015] The data standardization module standardizes key metadata fields of DICOM medical images and supports version control and audit tracking of key metadata fields, while also optimizing DICOM medical images.

[0016] The scoring module objectively assesses the quality of DICOM medical images based on a multi-dimensional image quality assessment model and calculates the corresponding comprehensive quality score.

[0017] The data storage module categorizes and stores DICOM medical images based on their overall quality scores.

[0018] The processing workflow recording module, based on a complete standardized processing workflow recording and operation traceability mechanism, records the entire process of DICOM medical images from entering the system to final output in detail, and adds structured processing tags to DICOM medical images.

[0019] Preferably, the data acquisition module includes an acquisition unit and a metadata extraction unit;

[0020] The acquisition unit includes an acquisition interface that supports multiple transmission protocols, enabling the acquisition of DICOM medical images from CT, MRI, X-ray equipment or PACS systems. It also supports local directory monitoring and cloud storage access, ensuring the unified aggregation of multi-source heterogeneous data.

[0021] The metadata extraction unit is based on a high-performance parsing engine developed according to the DICOM standard protocol. It automatically identifies and extracts key metadata fields in DICOM medical images, including patient information, examination information, equipment parameters, and image attributes.

[0022] The data acquisition module, based on the schema management mechanism, achieves compatibility and mapping of DICOM formats from different manufacturers, outputs structured data, and writes key metadata fields of DICOM medical images into a unified data lake or relational / non-relational database, providing basic data support for the construction of subsequent indexing systems.

[0023] Preferably, the index building module parses each DICOM medical image, identifies and records its unique identifier, and builds a hierarchical index structure based on the identifier of the DICOM medical image to reflect the hierarchical relationship of medical image examination;

[0024] The generated index information is written to two destinations: one is a structured database for persistent storage and complex query operations; the other is a Kafka message queue, which, as part of an event-driven architecture, triggers subsequent workflows such as image preprocessing and quality assessment to improve data access speed, promote collaborative work in a distributed environment, and make the entire DICOM medical image processing pipeline smoother and more intelligent.

[0025] Preferably, the data standardization module uses a unified metadata parsing and conversion tool to standardize the key metadata fields of each DICOM medical image, ensuring that this information is consistent and comparable in subsequent workflows;

[0026] The data standardization module supports version control and audit tracking of key metadata fields, ensuring data traceability and compliance requirements.

[0027] The data standardization module optimizes DICOM medical images through various methods, including noise reduction, enhancement, and normalization.

[0028] Preferably, the denoising includes using Spark's built-in image processing library, OpenCV library, and custom filters to remove noise interference in DICOM medical images and improve image clarity;

[0029] The enhancement includes using the contrast-limited adaptive histogram equalization (CLAHE) technique to enhance the local contrast of DICOM medical images, making details more prominent.

[0030] The normalization process involves adjusting pixel values ​​linearly or non-linearly to normalize DICOM medical images from different sources to the same grayscale range, eliminating color deviations caused by device differences and facilitating subsequent quantitative analysis.

[0031] Preferably, the scoring module first uses image processing algorithms to extract feature parameters of DICOM medical images, then sets weight coefficients corresponding to each feature parameter according to clinical needs and application scenarios, thereby improving the system's adaptability and flexibility while ensuring that the evaluation results meet actual medical standards. Finally, the feature parameters of DICOM medical images and their corresponding weight coefficients are input into a multi-dimensional image quality assessment model, and an objective quality assessment of DICOM medical images is performed based on the multi-dimensional image quality assessment model to calculate the corresponding comprehensive quality score.

[0032] Preferably, the scoring module constructs a multi-dimensional image quality assessment model using the following formula:

[0033] Q_score=α·SNR+β·Contrast+γ·Sharpness-δ·Artifacts;

[0034] Wherein, Q_score is the overall quality score of DICOM medical images, SNR is the signal-to-noise ratio of DICOM medical images, which measures the ratio between image signal intensity and background noise. A higher SNR value indicates clearer image details, Contrast is the contrast of DICOM medical images, representing the length of grayscale difference between different tissues or structures. Good contrast helps to distinguish lesions from normal tissues, Sharpness is the sharpness of DICOM medical images, reflecting the clarity of image edges. High sharpness can enhance the ability to identify small lesions, Artifacts are artifacts of DICOM medical images, quantifying the degree of image distortion caused by factors including equipment or patient movement, and α, β, γ, and δ are all weighting coefficients.

[0035] Preferably, the data storage module categorizes and stores DICOM medical images according to a comprehensive quality score to optimize storage costs and access efficiency.

[0036] Based on the overall quality score, high-quality, clinically valuable, and frequently accessed DICOM medical images are defined as "hot data" and stored on high-performance, low-latency standard storage media to ensure that they can be quickly accessed in subsequent workflows.

[0037] Based on the overall quality score, DICOM medical images of medium / low quality, low clinical value, low access frequency, or those requiring only long-term archiving are defined as "cold data" and uniformly archived to low-cost, low-frequency storage media with relatively high read latency.

[0038] Preferably, the processing flow recording module is based on a complete standardized processing flow recording and operation traceability mechanism, which records the entire process of DICOM medical images from entering the system to the final output in detail. This mechanism runs through key links including data acquisition, key metadata field extraction, index system construction, image optimization processing, image quality assessment and image classification and storage, ensuring that every step of the operation is traceable, the process can be restored, and the responsibility can be tracked.

[0039] The processing flow recording module introduces a multi-dimensional tagging system to assign structured processing tags to DICOM medical images. These processing tags cover:

[0040] Data source tags: equipment manufacturer, data collection method;

[0041] Processing stage tags: Collected, Metadata parsed, Optimization completed, Quality assessment completed, Classified and stored;

[0042] Quality rating labels: High quality / Medium quality / Low quality based on overall quality score;

[0043] Operation tags: Record the ID of the specific operator and the module being executed, supporting permission tracing;

[0044] Anomaly markers: used to identify special cases including failed optimization processes, overall quality scores below the threshold, and formatting errors;

[0045] Hot / cold attribute tags: Identified as hot data / cold data according to the classification storage strategy.

[0046] Preferably, this system adopts the Spark on Kubernetes construction mode, which makes full use of Spark's distributed computing capabilities and Kubernetes' elastic scheduling and resource management advantages. Spark applications can directly and dynamically request Pod resources on the Kubernetes cluster to execute Driver and Executor tasks, realizing true containerization and cloud-native big data processing.

[0047] Kubernetes provides flexible resource scheduling, service discovery, autoscaling, and high availability capabilities, enabling Spark jobs to automatically scale computing resources based on load, improving resource utilization and task execution efficiency. At the same time, Spark on Kubernetes supports features including fine-grained resource configuration, role-based access control (RBAC), image management, and log monitoring, enhancing the maintainability and security of big data applications in enterprise production environments.

[0048] (III) Beneficial Effects

[0049] Compared with existing technologies, the Spark-based DICOM image preprocessing and quality control system provided by this invention has the following advantages:

[0050] 1) This system fully leverages Spark's powerful distributed computing capabilities, memory acceleration mechanism, and task scheduling advantages. Combined with advanced image processing algorithms and standardized processes, it can efficiently process massive amounts of DICOM medical image files from different device manufacturers with inconsistent metadata formats. This significantly improves the acquisition efficiency, processing speed, and quality control of medical images. By integrating the unified access, parsing, indexing, optimization, quality assessment, and classified storage of DICOM medical images into a highly automated and scalable platform, the system effectively solves the problems of low efficiency, significant resource bottlenecks, and poor data compatibility that exist in traditional single-machine processing methods. It provides high-quality, structured medical image data support for hospitals, research institutions, and AI diagnostic platforms.

[0051] 2) This system supports unified acquisition and real-time processing of multi-source heterogeneous data, and has the capabilities of high concurrency, high fault tolerance and elastic scaling. It can complete batch processing or streaming analysis of large-scale medical images in a short time, greatly improving data flow efficiency.

[0052] 3) The system's built-in multi-dimensional image quality assessment model quantifies and scores multiple indicators such as signal-to-noise ratio, contrast, sharpness, and artifacts, thereby achieving automatic image quality assessment and ensuring that only data that meets the standards can enter the subsequent clinical analysis or AI model training stage.

[0053] 4) This system also constructs a complete metadata management system and a hot and cold data classification and storage mechanism. High-quality "hot data" is stored in high-performance, low-latency standard storage media for fast access, while medium / low-quality or only long-term archived "cold data" is stored in low-frequency storage media, thereby achieving a balance between cost optimization and access efficiency.

[0054] 5) This system introduces operation traceability and multi-dimensional tagging system throughout the entire process, recording key information such as the processing path, operator, timestamp, and quality status of each DICOM medical image, forming a complete audit log and traceability chain, meeting the high standards of the medical industry for data compliance, security and privacy protection.

[0055] 6) This system adopts this end-to-end standardized, intelligent, and visualized DICOM image preprocessing and quality control solution, which not only helps improve the efficiency of image management and diagnostic accuracy of medical institutions, but also lays a solid data foundation for the development of fields such as artificial intelligence-assisted diagnosis, telemedicine, and precision medicine research. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0057] Figure 1 This is a schematic diagram of the system of the present invention;

[0058] Figure 2 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0060] Spark-based DICOM image preprocessing and quality control system, such as Figure 1 and Figure 2 As shown, it includes a data acquisition module, an index building module, a data standardization module, a scoring module, a data storage module, and a processing flow recording module;

[0061] The data acquisition module collects DICOM medical images from the data source and automatically identifies and extracts key metadata fields from the DICOM medical images;

[0062] The index building module constructs a comprehensive and accurate index system based on the key metadata fields of DICOM medical images;

[0063] The data standardization module standardizes key metadata fields of DICOM medical images and supports version control and audit tracking of key metadata fields, while also optimizing DICOM medical images.

[0064] The scoring module objectively assesses the quality of DICOM medical images based on a multi-dimensional image quality assessment model and calculates the corresponding comprehensive quality score.

[0065] The data storage module categorizes and stores DICOM medical images based on their overall quality scores.

[0066] The processing workflow recording module, based on a complete standardized processing workflow recording and operation traceability mechanism, records the entire process of DICOM medical images from entering the system to final output in detail, and adds structured processing tags to DICOM medical images.

[0067] ① The data acquisition module includes an acquisition unit and a metadata extraction unit;

[0068] The acquisition unit includes an acquisition interface that supports multiple transmission protocols (including DICOM C-STORE, FTP, SFTP, HTTP / HTTPS, etc.) to acquire DICOM medical images from CT, MRI, X-ray equipment or PACS systems, and supports local directory monitoring and cloud storage access to ensure unified aggregation of multi-source heterogeneous data.

[0069] The metadata extraction unit develops a high-performance parsing engine based on DICOM standard protocols (such as DICOM 3.0) to automatically identify and extract key metadata fields from DICOM medical images, including patient information (Patient ID, name, age, etc.), examination information (StudyUID, SeriesUID, InstanceUID, Modality, Study Date, etc.), device parameters (Manufacturer, InstitutionName, etc.), and image attributes (Manufacturer, InstitutionName, etc.).

[0070] The data acquisition module, based on the schema management mechanism, achieves compatibility and mapping of DICOM formats from different manufacturers, outputs structured data (JSON or XML format), and writes key metadata fields of DICOM medical images into a unified data lake or relational / non-relational database, providing basic data support for the construction of subsequent indexing systems.

[0071] ② The index building module parses each DICOM medical image, identifies and records its unique identifier, and builds a hierarchical index structure based on the identifier of the DICOM medical image to reflect the hierarchical relationship of medical image examination (such as research level, sequence level, instance level, etc.);

[0072] The generated index information is written to two destinations: one is a structured database for persistent storage and complex query operations; the other is a Kafka message queue, which, as part of an event-driven architecture, triggers subsequent workflows such as image preprocessing and quality assessment to improve data access speed, promote collaborative work in a distributed environment, and make the entire DICOM medical image processing pipeline smoother and more intelligent.

[0073] ③ The data standardization module uses a unified metadata parsing and conversion tool to standardize the key metadata fields of each DICOM medical image, ensuring that this information is consistent and comparable in subsequent workflows;

[0074] The data standardization module supports version control and audit tracking of key metadata fields, ensuring data traceability and compliance requirements;

[0075] The data standardization module optimizes DICOM medical images through various methods, including denoising, enhancement, and normalization.

[0076] Specifically, denoising involves using Spark's built-in image processing library, OpenCV library, and custom filters (such as nonlocal mean filter NLM) to remove noise interference in DICOM medical images and improve image clarity.

[0077] Enhancement includes utilizing the contrast-limited adaptive histogram equalization (CLAHE) technique to enhance the local contrast of DICOM medical images, making details stand out more;

[0078] Normalization involves adjusting pixel values ​​linearly or non-linearly to normalize DICOM medical images from different sources to the same grayscale range, eliminating color deviations caused by device differences and facilitating subsequent quantitative analysis.

[0079] ④ The scoring module first uses image processing algorithms to extract feature parameters of DICOM medical images, and then sets the weight coefficients corresponding to each feature parameter according to clinical needs and application scenarios. This improves the system's adaptability and flexibility while ensuring that the evaluation results meet actual medical standards. Finally, the feature parameters of DICOM medical images and their corresponding weight coefficients are input into the multi-dimensional image quality assessment model. Based on the multi-dimensional image quality assessment model, the DICOM medical images are objectively evaluated, and the corresponding comprehensive quality score is calculated.

[0080] Specifically, the scoring module uses the following formula to construct a multi-dimensional image quality assessment model:

[0081] Q_score=α·SNR+β·Contrast+γ·Sharpness-δ·Artifacts;

[0082] Wherein, Q_score is the overall quality score of DICOM medical images, SNR is the signal-to-noise ratio of DICOM medical images, which measures the ratio between image signal intensity and background noise. A higher SNR value indicates clearer image details, Contrast is the contrast of DICOM medical images, representing the length of grayscale difference between different tissues or structures. Good contrast helps to distinguish lesions from normal tissues, Sharpness is the sharpness of DICOM medical images, reflecting the clarity of image edges. High sharpness can enhance the ability to identify small lesions, Artifacts are artifacts of DICOM medical images, quantifying the degree of image distortion caused by factors including equipment or patient movement, and α, β, γ, and δ are all weighting coefficients.

[0083] ⑤ The data storage module categorizes and stores DICOM medical images based on their overall quality score to optimize storage costs and access efficiency:

[0084] Based on the overall quality score, high-quality, clinically valuable, and frequently accessed DICOM medical images are defined as "hot data" and stored in high-performance, low-latency standard storage media (standard storage layer in object storage) to ensure that they can be quickly accessed in subsequent workflows.

[0085] Based on the overall quality score, DICOM medical images of medium / low quality, low clinical value, low access frequency, or those requiring only long-term archiving are defined as "cold data" and uniformly archived to low-cost, relatively high-latency low-frequency storage media (low-frequency storage layer in object storage).

[0086] ⑥ The processing flow recording module is based on a complete standardized processing flow recording and operation traceability mechanism. It records the entire process of DICOM medical images from entering the system to the final output in detail. This mechanism runs through key links including data acquisition, key metadata field extraction, index system construction, image optimization processing, image quality assessment and image classification and storage, ensuring that every operation is traceable, the process can be restored and the responsibility can be tracked.

[0087] The processing workflow recording module introduces a multi-dimensional tagging system to assign structured processing tags to DICOM medical images. These tags cover:

[0088] Data source tags: equipment manufacturer, acquisition method (e.g., PACS system, local upload, cloud synchronization, etc.);

[0089] Processing stage tags: Collected, Metadata parsed, Optimization completed, Quality assessment completed, Classified and stored;

[0090] Quality rating labels: High quality / Medium quality / Low quality based on overall quality score;

[0091] Operation tags: Record the ID of the specific operator and the module being executed, supporting permission tracing;

[0092] Anomaly markers: used to identify special cases including failed optimization processes, overall quality scores below the threshold, and formatting errors;

[0093] Hot / cold attribute tags: Identified as hot data / cold data according to the classification storage strategy.

[0094] In the technical solution of this application, the Spark-based DICOM image preprocessing and quality control system adopts the Spark on Kubernetes construction mode, which makes full use of Spark's distributed computing capabilities and Kubernetes' elastic scheduling and resource management advantages. Spark applications can directly and dynamically request Pod resources on the Kubernetes cluster to execute Driver and Executor tasks, realizing true containerization and cloud-native big data processing.

[0095] Kubernetes provides flexible resource scheduling, service discovery, autoscaling, and high availability capabilities, enabling Spark jobs to automatically scale computing resources based on load, improving resource utilization and task execution efficiency. At the same time, Spark on Kubernetes supports features including fine-grained resource configuration, role-based access control (RBAC), image management, and log monitoring, enhancing the maintainability and security of big data applications in enterprise production environments.

[0096] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A Spark-based DICOM image preprocessing and quality control system, characterized in that: It includes a data acquisition module, an index building module, a data standardization module, a scoring module, a data storage module, and a processing flow recording module; The data acquisition module collects DICOM medical images from the data source and automatically identifies and extracts key metadata fields from the DICOM medical images; The index building module constructs a comprehensive and accurate index system based on the key metadata fields of DICOM medical images; The data standardization module standardizes key metadata fields of DICOM medical images and supports version control and audit tracking of key metadata fields, while also optimizing DICOM medical images. The scoring module objectively assesses the quality of DICOM medical images based on a multi-dimensional image quality assessment model and calculates the corresponding comprehensive quality score. The data storage module categorizes and stores DICOM medical images based on their overall quality scores. The processing workflow recording module, based on a complete standardized processing workflow recording and operation traceability mechanism, records the entire process of DICOM medical images from entering the system to final output in detail, and adds structured processing tags to DICOM medical images.

2. The Spark-based DICOM image preprocessing and quality control system according to claim 1, characterized in that: The data acquisition module includes an acquisition unit and a metadata extraction unit; The acquisition unit includes an acquisition interface that supports multiple transmission protocols, enabling the acquisition of DICOM medical images from CT, MRI, X-ray equipment or PACS systems. It also supports local directory monitoring and cloud storage access, ensuring the unified aggregation of multi-source heterogeneous data. The metadata extraction unit is based on a high-performance parsing engine developed according to the DICOM standard protocol. It automatically identifies and extracts key metadata fields in DICOM medical images, including patient information, examination information, equipment parameters, and image attributes. The data acquisition module, based on the schema management mechanism, achieves compatibility and mapping of DICOM formats from different manufacturers, outputs structured data, and writes key metadata fields of DICOM medical images into a unified data lake or relational / non-relational database, providing basic data support for the construction of subsequent indexing systems.

3. The Spark-based DICOM image preprocessing and quality control system according to claim 1, characterized in that: The index building module parses each DICOM medical image, identifies and records its unique identifier, and builds a hierarchical index structure based on the identifier of the DICOM medical image to reflect the hierarchical relationship of medical image examination. The generated index information is written to two destinations: one is a structured database for persistent storage and complex query operations; the other is a Kafka message queue, which, as part of an event-driven architecture, triggers subsequent workflows such as image preprocessing and quality assessment to improve data access speed, promote collaborative work in a distributed environment, and make the entire DICOM medical image processing pipeline smoother and more intelligent.

4. The Spark-based DICOM image preprocessing and quality control system according to claim 1, characterized in that: The data standardization module uses a unified metadata parsing and conversion tool to standardize the key metadata fields of each DICOM medical image, ensuring that this information is consistent and comparable in subsequent workflows. The data standardization module supports version control and audit tracking of key metadata fields, ensuring data traceability and compliance requirements. The data standardization module optimizes DICOM medical images through various methods, including noise reduction, enhancement, and normalization.

5. The Spark-based DICOM image preprocessing and quality control system according to claim 4, characterized in that: The denoising process includes using Spark's built-in image processing library, OpenCV library, and custom filters to remove noise interference in DICOM medical images and improve image clarity. The enhancement includes using the contrast-limited adaptive histogram equalization (CLAHE) technique to enhance the local contrast of DICOM medical images, making details more prominent. The normalization process involves adjusting pixel values ​​linearly or non-linearly to normalize DICOM medical images from different sources to the same grayscale range, eliminating color deviations caused by device differences and facilitating subsequent quantitative analysis.

6. The Spark-based DICOM image preprocessing and quality control system according to claim 1, characterized in that: The scoring module first uses image processing algorithms to extract feature parameters of DICOM medical images, then sets the weight coefficients corresponding to each feature parameter according to clinical needs and application scenarios. This improves the system's adaptability and flexibility while ensuring that the evaluation results meet actual medical standards. Finally, the feature parameters of DICOM medical images and their corresponding weight coefficients are input into a multi-dimensional image quality assessment model. Based on the multi-dimensional image quality assessment model, an objective quality assessment of DICOM medical images is performed, and the corresponding comprehensive quality score is calculated.

7. The Spark-based DICOM image preprocessing and quality control system according to claim 6, characterized in that: The scoring module uses the following formula to construct a multi-dimensional image quality assessment model: Q_score=α·SNR+β·Contrast+γ·Sharpness-δ·Artifacts; Wherein, Q_score is the overall quality score of DICOM medical images, SNR is the signal-to-noise ratio of DICOM medical images, which measures the ratio between image signal intensity and background noise. A higher SNR value indicates clearer image details, Contrast is the contrast of DICOM medical images, representing the length of grayscale difference between different tissues or structures. Good contrast helps to distinguish lesions from normal tissues, Sharpness is the sharpness of DICOM medical images, reflecting the clarity of image edges. High sharpness can enhance the ability to identify small lesions, Artifacts are artifacts of DICOM medical images, quantifying the degree of image distortion caused by factors including equipment or patient movement, and α, β, γ, and δ are all weighting coefficients.

8. The Spark-based DICOM image preprocessing and quality control system according to claim 1, characterized in that: The data storage module categorizes and stores DICOM medical images based on a comprehensive quality score to optimize storage costs and access efficiency. Based on the overall quality score, high-quality, clinically valuable, and frequently accessed DICOM medical images are defined as "hot data" and stored on high-performance, low-latency standard storage media to ensure that they can be quickly accessed in subsequent workflows. Based on the overall quality score, DICOM medical images of medium / low quality, low clinical value, low access frequency, or those only requiring long-term archiving are defined as "cold data" and uniformly archived to low-cost, low-frequency storage media with relatively high read latency.

9. The Spark-based DICOM image preprocessing and quality control system according to claim 1, characterized in that: The processing flow recording module is based on a complete standardized processing flow recording and operation traceability mechanism. It records the entire process of DICOM medical images from entering the system to the final output in detail. This mechanism runs through key links including data acquisition, key metadata field extraction, index system construction, image optimization processing, image quality assessment and image classification and storage, ensuring that every operation is traceable, the process can be restored and the responsibility can be tracked. The processing flow recording module introduces a multi-dimensional tagging system to assign structured processing tags to DICOM medical images. These processing tags cover: Data source tags: equipment manufacturer, data collection method; Processing stage tags: Collected, Metadata parsed, Optimization completed, Quality assessment completed, Classified and stored; Quality rating labels: High quality / Medium quality / Low quality based on overall quality score; Operation tags: Record the ID of the specific operator and the module being executed, supporting permission tracing; Anomaly markers: used to identify special cases including failed optimization processes, overall quality scores below the threshold, and formatting errors; Hot / cold attribute tags: Identified as hot data / cold data according to the classification storage strategy.

10. The Spark-based DICOM image preprocessing and quality control system according to any one of claims 1-9, characterized in that: This system adopts the Spark on Kubernetes construction mode, which makes full use of Spark's distributed computing capabilities and Kubernetes' elastic scheduling and resource management advantages. Spark applications can directly and dynamically request Pod resources on the Kubernetes cluster to execute Driver and Executor tasks, realizing true containerization and cloud-native big data processing. Kubernetes provides flexible resource scheduling, service discovery, autoscaling, and high availability capabilities, enabling Spark jobs to automatically scale computing resources based on load, improving resource utilization and task execution efficiency. At the same time, Spark on Kubernetes supports features including fine-grained resource configuration, role-based access control (RBAC), image management, and log monitoring, enhancing the maintainability and security of big data applications in enterprise production environments.

Images