A big data image processing method of a computer
By adopting a five-layer architecture of distributed acquisition, hierarchical preprocessing, parallel feature extraction, hybrid storage, and intelligent analysis, the problem of insufficient computing power and high data pressure in traditional image processing methods is solved, achieving efficient and flexible image processing, improving computing power utilization and storage efficiency, and supporting rapid adaptation of multi-source heterogeneous images.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUANGSHAN UNIV
- Filing Date
- 2026-02-09
- Publication Date
- 2026-06-05
AI Technical Summary
In existing technologies, most traditional image processing methods are based on single-machine computing power, which has problems such as insufficient computing power, high pressure on data transmission and storage, poor preprocessing targeting, imbalance between feature extraction efficiency and accuracy, and insufficient flexibility in analysis and application.
It adopts a five-layer architecture of distributed acquisition, hierarchical preprocessing, parallel feature extraction, hybrid storage, and intelligent analysis and application. Combined with the computing power collaboration between the cloud and the edge, it realizes parallel, lightweight, and intelligent processing throughout the entire process, including distributed image data acquisition, hierarchical preprocessing, parallel feature extraction, hybrid storage, and distributed intelligent analysis.
It improves computing power utilization, optimizes transmission and storage costs, balances processing accuracy and real-time performance, supports rapid adaptation to multi-source heterogeneous image formats and different business scenarios, and has strong scalability and maintainability.
Abstract
Description
Technical Field
[0001] This application relates to the technical field of big data image processing, and in particular to a computer-based big data image processing method. Background Technology
[0002] With the popularization of computer vision technology and the widespread application of image acquisition equipment (smart cameras, drones, medical imaging instruments, etc.), image data has experienced explosive growth, forming typical big data characteristics: large data scale (PB level or even EB level), high dimensionality (high resolution, multi-channel), multi-source heterogeneity (diverse formats, scattered acquisition scenarios), and high real-time requirements (such as industrial quality inspection and intelligent monitoring requiring millisecond-level response).
[0003] However, in practical applications, most existing traditional image processing methods are mainly based on single-machine computing power and adopt the "serial acquisition-centralized preprocessing-single feature extraction-local storage" model. This model suffers from technical defects such as insufficient computing power, high pressure on data transmission and storage, poor preprocessing targeting, imbalance between feature extraction efficiency and accuracy, and insufficient flexibility in analysis and application.
[0004] Therefore, those skilled in the art provide a computer-based big data image processing method to solve the problems mentioned in the background art. Summary of the Invention
[0005] To address the technical shortcomings of most traditional image processing methods mentioned in the background, which are mainly based on single-machine computing power and adopt a "serial acquisition-centralized preprocessing-single feature extraction-local storage" model, resulting in insufficient computing power, high pressure on data transmission and storage, poor preprocessing targeting, imbalance between feature extraction efficiency and accuracy, and insufficient flexibility in analysis and application, this application provides a computer-based big data image processing method.
[0006] This application provides a computer-based big data image processing method using the following technical solution: based on a five-layer architecture of "distributed acquisition - hierarchical preprocessing - parallel feature extraction - hybrid storage - intelligent analysis and application", combined with cloud and edge computing power collaboration and heterogeneous acceleration technology, it achieves parallel, lightweight, and intelligent processing throughout the entire process, specifically including the following steps:
[0007] 1. Distributed Image Data Acquisition and Normalization Access: Deploy distributed acquisition terminals supporting multiple protocols such as HTTP / HTTPS, MQTT, and RTSP to perform format conversion, resolution unification, and pixel depth standardization on multi-source heterogeneous image data in DICOM, TIFF, JPG, and PNG formats, generating a standard digital image dataset; adopt a "edge caching + cloud synchronization" transmission strategy, using incremental transmission and breakpoint resume technology to synchronize data to the cloud, and utilize data sharding technology to achieve distributed storage of the dataset;
[0008] 2. Layered Image Preprocessing: At the edge, millisecond-level lightweight preprocessing is performed, including redundant frame filtering, lightweight noise suppression, and adaptive scaling. Redundant frames are removed using inter-frame difference, and noise is suppressed using fast median filtering or adaptive mean filtering. At the cloud, refined preprocessing is performed, including precise noise removal, multi-dimensional image enhancement, geometric correction and registration, and Region of Interest (ROI) extraction. Precise noise removal uses Gaussian bilateral filtering, non-local mean filtering, or wavelet denoising algorithms. Image enhancement includes brightness / contrast adjustment, edge enhancement, and blur restoration. Geometric correction is achieved through perspective transformation or affine transformation.
[0009] 3. Parallel Image Feature Extraction and Dimensionality Reduction: Low-level visual features are extracted in parallel using a Spark distributed cluster, and high-level semantic features are extracted in parallel on a GPU / TPU heterogeneous acceleration cluster based on a lightweight deep learning model. The ultra-large-scale dataset adopts a "distributed model training + distributed feature extraction" mode. The dimensionality of the fused high-dimensional features is reduced by traditional algorithms such as PCA, LDA, and t-SNE or deep learning algorithms such as AE, VAE, and CAE to generate lightweight feature vectors. After weighted fusion or feature concatenation, Z-Score or Min-Max standardization is performed.
[0010] 4. Hybrid Data Storage: A hybrid storage architecture of "distributed file system + object storage + vector database + relational database + time-series database" is adopted. HDFS stores the frequently accessed raw image hot data, OSS / S3 object stores the infrequently accessed raw image cold data, Milvus or Pinecone distributed vector database stores feature vectors, MySQL or PostgreSQL stores metadata, and InfluxDB or TimescaleDB stores processing logs; an automatic cold and hot data migration and expired data deletion lifecycle management mechanism is established;
[0011] 5. Distributed Intelligent Analysis and Application: Deploy lightweight deep learning models based on heterogeneous accelerated clusters, perform parallel inference on feature vectors, and output real-time analysis results; perform distributed data mining based on SparkMLlib or TensorFlowDistributed frameworks; and achieve online iterative optimization of models through analysis result feedback.
[0012] Preferably, the distributed acquisition terminal in step 1 includes a smart camera, an edge gateway, and an image acquisition device, and the standard digital image format is an 8-bit / 16-bit grayscale image or a 24-bit RGB color image.
[0013] Preferably, in step 2, the amount of data after filtering redundant frames at the edge is reduced by 60% to 90%, and the processing latency is controlled at the millisecond level; cloud ROI extraction is achieved through threshold segmentation, region growing, or contour detection algorithms.
[0014] Preferably, in step 3, the low-level visual features include color histograms, GLCM texture features, and rectangular features, while the high-level semantic features are extracted using lightweight versions of MobileNet, ShuffleNet, or ResNet models.
[0015] Preferably, the results of parallel inference in step 5 include target detection, classification, and segmentation information; distributed data mining includes clustering, association rule mining, and anomaly detection; and real-time processing latency is controlled within 100ms.
[0016] In summary, this application includes the following beneficial technical effects:
[0017] 1. Improved computing power utilization: The combination of the "cloud + edge" collaborative architecture and distributed parallel computing enables computing resources to be allocated on demand. Compared with single-machine processing, the overall processing efficiency is improved by 3 to 5 times, which can support the daily processing of PB-level image data.
[0018] 2. Optimization of transmission and storage costs: Edge preprocessing eliminates a large amount of redundant data, and incremental transmission reduces bandwidth usage by more than 50%; the hybrid storage architecture reduces storage costs by 30% to 40%, while also meeting the needs of fast retrieval and massive archiving.
[0019] 3. Balance between processing accuracy and real-time performance: By integrating low-level visual features and high-level semantic features, the target recognition accuracy can reach over 95%. Lightweight preprocessing at the edge and parallel inference in the cloud ensure that the real-time processing latency is ≤100ms, meeting the high real-time business requirements.
[0020] 4. Wide scene adaptability: It supports multi-source heterogeneous image formats and intelligent monitoring, medical imaging, industrial quality inspection and remote sensing interpretation in different business scenarios. By replacing the corresponding acquisition terminal, preprocessing algorithm and feature extraction model, it can be quickly adapted to new scenarios.
[0021] 5. Strong scalability and maintainability: The distributed architecture supports horizontal scaling of collection nodes and computing nodes, metadata and log storage enables full-process traceability, and the online model iteration mechanism can complete upgrades without interrupting business operations. Detailed Implementation
[0022] The described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.
[0023] This application discloses a computer-based big data image processing method. Based on a five-layer architecture of "distributed acquisition - hierarchical preprocessing - parallel feature extraction - hybrid storage - intelligent analysis and application," and combining cloud and edge computing power collaboration and heterogeneous acceleration technology, it achieves fully parallel, lightweight, and intelligent processing. Specifically, it includes the following steps:
[0024] 1. Distributed Image Data Acquisition and Normalization Access: Deploy distributed acquisition terminals supporting multiple protocols such as HTTP / HTTPS, MQTT, and RTSP to perform format conversion, resolution unification, and pixel depth standardization on multi-source heterogeneous image data in DICOM, TIFF, JPG, and PNG formats, generating a standard digital image dataset; adopt a "edge caching + cloud synchronization" transmission strategy, using incremental transmission and breakpoint resume technology to synchronize data to the cloud, and utilize data sharding technology to achieve distributed storage of the dataset;
[0025] 2. Layered Image Preprocessing: At the edge, millisecond-level lightweight preprocessing is performed, including redundant frame filtering, lightweight noise suppression, and adaptive scaling. Redundant frames are removed using inter-frame difference, and noise is suppressed using fast median filtering or adaptive mean filtering. At the cloud, refined preprocessing is performed, including precise noise removal, multi-dimensional image enhancement, geometric correction and registration, and Region of Interest (ROI) extraction. Precise noise removal uses Gaussian bilateral filtering, non-local mean filtering, or wavelet denoising algorithms. Image enhancement includes brightness / contrast adjustment, edge enhancement, and blur restoration. Geometric correction is achieved through perspective transformation or affine transformation.
[0026] 3. Parallel Image Feature Extraction and Dimensionality Reduction: Low-level visual features are extracted in parallel using a Spark distributed cluster, and high-level semantic features are extracted in parallel on a GPU / TPU heterogeneous acceleration cluster based on a lightweight deep learning model. The ultra-large-scale dataset adopts a "distributed model training + distributed feature extraction" mode. The dimensionality of the fused high-dimensional features is reduced by traditional algorithms such as PCA, LDA, and t-SNE or deep learning algorithms such as AE, VAE, and CAE to generate lightweight feature vectors. After weighted fusion or feature concatenation, Z-Score or Min-Max standardization is performed.
[0027] 4. Hybrid Data Storage: A hybrid storage architecture of "distributed file system + object storage + vector database + relational database + time-series database" is adopted. HDFS stores the frequently accessed raw image hot data, OSS / S3 object stores the infrequently accessed raw image cold data, Milvus or Pinecone distributed vector database stores feature vectors, MySQL or PostgreSQL stores metadata, and InfluxDB or TimescaleDB stores processing logs; an automatic cold and hot data migration and expired data deletion lifecycle management mechanism is established;
[0028] 5. Distributed Intelligent Analysis and Application: Deploy lightweight deep learning models based on heterogeneous accelerated clusters, perform parallel inference on feature vectors, and output real-time analysis results; perform distributed data mining based on SparkMLlib or TensorFlowDistributed frameworks; and achieve online iterative optimization of models through analysis result feedback.
[0029] In this application, the distributed acquisition terminal in step 1 includes a smart camera, an edge gateway, and an image acquisition device, and the standard digital image format is an 8-bit / 16-bit grayscale image or a 24-bit RGB color image.
[0030] In this application, the amount of data after filtering redundant frames at the edge in step 2 is reduced by 60% to 90%, and the processing latency is controlled at the millisecond level; cloud ROI extraction is achieved through threshold segmentation, region growing, or contour detection algorithms.
[0031] In this application, the low-level visual features in step 3 include color histograms, GLCM texture features, and rectangular features, while the high-level semantic features are extracted using lightweight versions of MobileNet, ShuffleNet, or ResNet models.
[0032] In this application, the results of parallel inference in step 5 include target detection, classification, and segmentation information; distributed data mining includes clustering, association rule mining, and anomaly detection; and real-time processing latency is controlled within 100ms.
[0033] All standard parts used in this invention can be purchased from the market, and irregular parts can be customized according to the description and drawings. The specific installation methods of each part adopt conventional methods such as bolts, rivets, and welding that are mature in the prior art. The machinery, parts and equipment adopt conventional models in the prior art. In addition, the circuit installation adopts conventional installation methods in the prior art, which will not be described in detail here.
[0034] Secondly: The accompanying drawings of the embodiments disclosed in this invention only involve the structures involved in the embodiments disclosed in this invention. Other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of this invention can be combined with each other.
[0035] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A computer-based method for processing large amounts of images, characterized in that, Based on a five-layer architecture of "distributed acquisition - hierarchical preprocessing - parallel feature extraction - hybrid storage - intelligent analysis and application", combined with cloud and edge computing power collaboration and heterogeneous acceleration technology, the entire process is parallelized, lightweight, and intelligently processed, specifically including the following steps: (1) Distributed image data acquisition and normalization access: Deploy distributed acquisition terminals that support multiple protocols such as HTTP / HTTPS, MQTT, and RTSP to perform format conversion, resolution unification, and pixel depth standardization on multi-source heterogeneous image data in DICOM, TIFF, JPG, and PNG formats to generate standard digital image datasets; Adopt the transmission strategy of "edge caching + cloud synchronization" to synchronize data to the cloud through incremental transmission and breakpoint resume transmission technology, and use data sharding technology to realize distributed storage of datasets; (2) Layered image preprocessing: The edge performs millisecond-level lightweight preprocessing, including redundant frame filtering, lightweight noise suppression, and adaptive scaling. Redundant frames are removed based on the inter-frame difference method, and noise is suppressed by fast median filtering or adaptive mean filtering. The cloud performs fine preprocessing, including precise noise removal, multi-dimensional image enhancement, geometric correction and registration, and region of interest (ROI) extraction. Precise noise removal uses Gaussian bilateral filtering, non-local mean filtering, or wavelet denoising algorithms. Image enhancement includes brightness / contrast adjustment, edge enhancement, and blur restoration. Geometric correction is achieved through perspective transformation or affine transformation. (3) Parallel image feature extraction and dimensionality reduction: Low-level visual features are extracted in parallel through Spark distributed clusters, and high-level semantic features are extracted in parallel on GPU / TPU heterogeneous acceleration clusters based on lightweight deep learning models. The ultra-large-scale dataset adopts the "distributed model training + distributed feature extraction" mode. The high-dimensional features after fusion are reduced in dimensionality by traditional algorithms such as PCA, LDA, and t-SNE or deep learning algorithms such as AE, VAE, and CAE to generate lightweight feature vectors. After weighted fusion or feature concatenation, Z-Score or Min-Max standardization is performed. (4) Hybrid data storage: A hybrid storage architecture of "distributed file system + object storage + vector database + relational database + time series database" is adopted. HDFS stores the high-frequency access raw image hot data, OSS / S3 object stores the low-frequency access raw image cold data, Milvus or Pinecone distributed vector database stores feature vectors, MySQL or PostgreSQL stores metadata, and InfluxDB or TimescaleDB stores processing logs; an automatic cold and hot data migration and expired data deletion lifecycle management mechanism is established; (5) Distributed intelligent analysis and application: Deploy lightweight deep learning models based on heterogeneous accelerated clusters, perform parallel inference on feature vectors, and output real-time analysis results; perform distributed data mining based on SparkMLlib or TensorFlowDistributed frameworks; and achieve online iterative optimization of models through feedback of analysis results.
2. The computer big data image processing method according to claim 1, characterized in that, The distributed acquisition terminal in step (1) includes a smart camera, an edge gateway, and an image acquisition device. The standard digital image format is an 8-bit / 16-bit grayscale image or a 24-bit RGB color image.
3. The computer big data image processing method according to claim 1, characterized in that, In step (2), the amount of data after filtering redundant frames at the edge is reduced by 60% to 90%, and the processing latency is controlled at the millisecond level; cloud ROI extraction is achieved through threshold segmentation, region growing or contour detection algorithms.
4. The computer big data image processing method according to claim 1, characterized in that, In step (3), the underlying visual features include color histogram, GLCM texture features, and rectangular features, while the high-level semantic features are extracted using lightweight versions of MobileNet, ShuffleNet, or ResNet models.
5. The computer big data image processing method according to claim 1, characterized in that, The results of parallel inference in step (5) include target detection, classification, and segmentation information. Distributed data mining includes clustering, association rule mining, and anomaly detection. Real-time processing latency is controlled within 100ms.