Image and data analysis processing method and device for oversized file and storage medium

Abstract

The invention provides an image and data analysis processing method for a super-large file and electronic equipment and a storage medium thereof, and the method comprises the steps: segmenting the super-large file into a plurality of sub-files through an intelligent segmentation algorithm, building an index, and constructing a distributed storage network based on edge computing and cloud computing; according to the method, distributed storage and parallel processing of sub-files are achieved, GPU accelerated image processing and data analysis algorithms are utilized, the deep learning technology is combined, efficient processing and intelligent analysis are conducted on the sub-files, finally, processing results of storage nodes are summarized through the data fusion technology, the processing flow is optimized, and the overall efficiency and quality are improved. According to the technical scheme, the problems of resource bottleneck, low processing speed and low analysis precision in ultra-large file processing are effectively solved, large-scale data processing requirements are met, and low delay, high availability and safety of data processing are ensured.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and more specifically, to an image and data analysis and processing method, device and storage medium for ultra-large files. Background Art

[0002] With the rapid development of information technology, the scale of images and data is becoming increasingly large. Traditional processing methods are unable to cope with extremely large files. In the current field of information technology, image and data analysis and processing technology for extremely large files is becoming more and more important. With the explosive growth of data volume, especially in the fields of medical imaging, satellite imaging, video surveillance and Internet multimedia, the demand for effective processing of extremely large files is becoming increasingly urgent.

[0003] The existing technology has the problems of low efficiency in processing very large files, poor security, and inflexible resource scheduling. Summary of the invention

[0004] In order to overcome the problems of low efficiency, poor security and inflexible resource scheduling in the existing technology for ultra-large file processing, the present invention designs an image and data analysis and processing method, device and storage medium for ultra-large files, which can effectively solve the technical problems of Shanshu.

[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0006] A method for analyzing and processing images and data of super-large files, using an intelligent segmentation algorithm to automatically segment a super-large file into multiple sub-files, and to create an index for each of the sub-files;

[0007] Build a distributed storage network based on edge computing and cloud computing, including edge storage nodes and cloud storage nodes;

[0008] According to the file index, sub-files are allocated to different storage nodes, and an adaptive load balancing algorithm is used to dynamically schedule computing resources to achieve parallel processing;

[0009] On each of the storage nodes, the sub-files are processed using GPU-accelerated image processing and data analysis algorithms, and deep learning technology is used to intelligently analyze the processing results;

[0010] The processing results of the storage nodes are aggregated, the final analysis results are obtained through data fusion technology, and the processing flow is optimized according to the analysis results.

[0011] Preferably, the method of using an intelligent segmentation algorithm to automatically segment a large file into multiple sub-files and creating an index for each sub-file includes the following specific steps:

[0012] Analyze the structure and content of the super-large file to determine a suitable splitting reference point;

[0013] According to a preset segmentation rule, the super large file is segmented into a plurality of sub-files;

[0014] Unique index information is generated for each sub-file, and the index information at least includes a starting position, an ending position, a data type, and associated metadata of the sub-file.

[0015] Preferably, the specific steps of constructing a distributed storage network based on edge computing and cloud computing include:

[0016] Selecting geographically dispersed edge storage nodes to ensure low latency and high availability of data processing;

[0017] Deploy the cloud storage nodes with powerful computing and storage capabilities to support large-scale data processing needs;

[0018] A communication mechanism is established between the edge storage node and the cloud storage node to ensure data synchronization and task coordination.

[0019] Preferably, the specific steps of allocating sub-files to different storage nodes according to the file index and using an adaptive load balancing algorithm to dynamically schedule computing resources to achieve parallel processing include:

[0020] According to the index information of the sub-file, real-time monitoring of the load conditions of the edge storage node and the cloud storage node;

[0021] Applying an adaptive load balancing algorithm to dynamically distribute the sub-files to the storage nodes with lighter loads;

[0022] The computing resources are scheduled to ensure that the storage nodes can process the allocated sub-files in parallel.

[0023] Preferably, the image processing and data analysis algorithm accelerated by GPU includes:

[0024] Deploy a GPU-accelerated image processing unit on each of the storage nodes to increase the image processing speed;

[0025] Using the parallel processing capability of GPU to efficiently process the image data in the sub-file;

[0026] A data analysis algorithm is implemented to perform in-depth analysis on the data in the subfiles.

[0027] Preferably, the intelligent analysis of the processing results using deep learning technology includes:

[0028] Train deep learning models for intelligent analysis of image processing and data analysis results;

[0029] Deploy a deep learning model on each of the storage nodes to implement real-time analysis of the sub-file processing results;

[0030] Adjust the parameters of the deep learning model based on the analysis results to optimize the analysis accuracy.

[0031] Preferably, the specific steps of aggregating the processing results of the storage nodes and obtaining the final analysis results by using data fusion technology include:

[0032] Collect the processing results of the storage nodes to ensure the integrity and consistency of the data;

[0033] Apply data fusion algorithm to integrate the processing results of different storage nodes to obtain the final analysis results;

[0034] Based on the final analysis results, feedback is provided to optimize the processing flow and improve overall processing efficiency and quality.

[0035] An electronic device, comprising:

[0036] A memory storing executable program code;

[0037] a processor coupled to the memory;

[0038] The processor calls the executable program code stored in the memory to execute the image and data analysis and processing method for ultra-large files as described above.

[0039] A computer storage medium stores computer instructions, which, when called, are used to execute the above-mentioned image and data analysis and processing method for very large files.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows: firstly, the present invention automatically divides a super large file into multiple sub-files by adopting an intelligent segmentation algorithm, and establishes an index for each sub-file, which significantly improves the efficiency of file processing. The intelligent segmentation algorithm can determine the most appropriate segmentation reference point according to the structure and content of the file, thereby ensuring that the segmented sub-files are easy to process and maintain the integrity of the data. At the same time, the index information established for each sub-file facilitates subsequent data processing and analysis. Secondly, a distributed storage network based on edge computing and cloud computing is constructed to achieve fast access to and efficient processing of data. The deployment of edge storage nodes ensures low latency and high availability of data processing, while cloud storage nodes provide It has powerful computing and storage capabilities and supports large-scale data processing needs. This distributed storage network not only improves data security, but also realizes dynamic scheduling and parallel processing of computing resources through adaptive load balancing algorithms, further improving processing efficiency. Finally, this technical solution also uses GPU-accelerated image processing and data analysis algorithms, as well as deep learning technology to perform intelligent analysis of processing results. The parallel processing capability of GPU significantly improves the speed of image processing and data analysis, while the deep learning model can perform real-time, accurate and intelligent analysis of processing results. This intelligent processing method not only improves the accuracy of analysis, but also continuously optimizes the processing process through a feedback mechanism, thereby improving overall processing efficiency and quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived based on the provided drawings without paying any creative work.

[0042] Figure 1 A step diagram of an image and data analysis and processing method for very large files; DETAILED DESCRIPTION

[0043] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0044] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0045] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0046] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0047] Example

[0048] A method for image and data analysis and processing of very large files, such as Figure 1 As shown,

[0049] An intelligent segmentation algorithm is used to automatically segment oversized files into multiple sub-files, and an index is created for each of the sub-files. In-depth structural and content analysis is performed on oversized files to identify the logical structure, data type, and key content within the file to determine the appropriate segmentation reference point. In combination with preset segmentation rules, such as file size, an intelligent segmentation strategy is formulated. According to the segmentation strategy, oversized files are segmented into multiple sub-files, each of which maintains data integrity and independence. Unique index information is generated for each sub-file, which includes the sub-file's identifier, location information, data type, content summary, etc., to facilitate subsequent management and processing.

[0050] Build a distributed storage network based on edge computing and cloud computing, including edge storage nodes and cloud storage nodes;

[0051] Based on factors such as geographical location, network bandwidth, storage capacity, and computing power, select and deploy edge storage nodes and cloud storage nodes, establish communication links between edge storage nodes and cloud storage nodes, ensure that data can be transmitted efficiently and securely between nodes, and formulate storage strategies based on the index information of sub-files and the characteristics of storage nodes, including data redundancy, backup, and recovery mechanisms.

[0052] According to the file index, sub-files are allocated to different storage nodes, and an adaptive load balancing algorithm is used to dynamically schedule computing resources to achieve parallel processing;

[0053] The distribution of sub-files can be to edge storage nodes or cloud storage nodes, depending on a variety of factors, including but not limited to the following:

[0054] Data processing needs: If data processing requires a quick response and the data volume is not large, it may be allocated to edge storage nodes first to reduce latency.

[0055] Storage and computing resources,If the resources of the edge storage node are insufficient to process a sub-file,,then the sub-file may be assigned to a cloud storage node with,richer resources.

[0056] Network conditions: If network conditions permit and the cloud storage node can provide more efficient processing capabilities, the sub-file may be assigned to the cloud storage node.

[0057] Data security and privacy requirements: Sensitive data may need to be stored on specific storage nodes to meet security and privacy requirements.

[0058] According to the adaptive load balancing algorithm, the system will distribute sub-files according to the following steps:

[0059] Monitor the real-time load of each storage node, including edge storage nodes and cloud storage nodes.

[0060] Based on the preset allocation strategy and the current load situation, it is dynamically decided whether to allocate the sub-file to the edge storage node or the cloud storage node.

[0061] The goal is to balance the load across the network, optimize resource utilization, and enable efficient data processing.

[0062] On each of the storage nodes, the sub-files are processed using GPU-accelerated image processing and data analysis algorithms, and deep learning technology is used to intelligently analyze the processing results;

[0063] Allocate GPU resources to each storage node to ensure that image processing and data analysis tasks can fully utilize the parallel processing capabilities of the GPU. Deploy GPU-accelerated image processing algorithms on storage nodes to efficiently process image data in sub-files. Deploy data analysis algorithms on storage nodes to conduct in-depth analysis of non-image data in sub-files and extract key information.

[0064] The processing results of the storage nodes are aggregated, the final analysis results are obtained through data fusion technology, and the processing flow is optimized according to the analysis results.

[0065] The method of using an intelligent segmentation algorithm to automatically segment a large file into multiple sub-files and creating an index for each sub-file includes the following specific steps:

[0066] Analyze the structure and content of the super-large file to determine a suitable splitting reference point;

[0067] According to a preset segmentation rule, such as a file size threshold, data block integrity or a specific content mark, the super large file is segmented into a plurality of sub-files;

[0068] Unique index information is generated for each sub-file, and the index information at least includes a starting position, an ending position, a data type, and associated metadata of the sub-file.

[0069] The specific steps of constructing a distributed storage network based on edge computing and cloud computing include:

[0070] Selecting geographically dispersed edge storage nodes to ensure low latency and high availability of data processing;

[0071] Deploy the cloud storage nodes with powerful computing and storage capabilities to support large-scale data processing needs;

[0072] A communication mechanism is established between the edge storage node and the cloud storage node to ensure data synchronization and task coordination.

[0073] The specific steps of allocating sub-files to different storage nodes according to the file index and using an adaptive load balancing algorithm to dynamically schedule computing resources to achieve parallel processing include:

[0074] According to the index information of the sub-file, real-time monitoring of the load conditions of the edge storage node and the cloud storage node;

[0075] Applying an adaptive load balancing algorithm to dynamically distribute the sub-files to the storage nodes with lighter loads;

[0076] The computing resources are scheduled to ensure that the storage nodes can process the allocated sub-files in parallel.

[0077] The image processing and data analysis algorithms accelerated by GPU include:

[0078] Deploy a GPU-accelerated image processing unit on each of the storage nodes to increase the image processing speed;

[0079] Using the parallel processing capability of GPU to efficiently process the image data in the sub-file;

[0080] A data analysis algorithm is implemented to perform in-depth analysis on the data in the subfiles.

[0081] The use of deep learning technology to intelligently analyze the processing results includes:

[0082] Train deep learning models for intelligent analysis of image processing and data analysis results;

[0083] Deploy a deep learning model on each of the storage nodes to implement real-time analysis of the sub-file processing results;

[0084] Adjust the parameters of the deep learning model based on the analysis results to optimize the analysis accuracy.

[0085] Deep learning models are trained using large, annotated datasets to recognize complex patterns and features in images and data.

[0086] Convolutional Neural Networks (CNN) can be used to process image data, Recurrent Neural Networks (RNN) or Long Short-Term Memory Networks (LSTM) can be used to process sequence data, and other types of neural networks can be used to process different data types.

[0087] Through training, the parameters learned by the model can capture high-level features in the data and provide powerful feature representation for subsequent analysis.

[0088] Deploy a trained deep learning model on each storage node, allowing each node to independently perform real-time analysis on sub-files.

[0089] This distributed deployment method improves the speed and efficiency of analysis because data does not need to be transmitted across the network to a central node for processing, reducing latency and bandwidth consumption.

[0090] At the same time, this also improves the scalability and fault tolerance of the system, because each node can work independently, and even if a problem occurs in a node, it will not affect the operation of the entire system.

[0091] By monitoring and evaluating the model's analytical results, the model's parameters can be fine-tuned to optimize the model's performance and accuracy.

[0092] Online learning or incremental learning methods can be used to dynamically update the model based on new data so that it can adapt to changes in data distribution.

[0093] By adjusting the model parameters, the model's robustness to outliers can be improved, overfitting can be reduced, and the model's generalization ability can be improved.

[0094] The specific steps of aggregating the processing results of the storage nodes and obtaining the final analysis results through data fusion technology include:

[0095] Collect the processing results of the storage nodes to ensure the integrity and consistency of the data;

[0096] Apply data fusion algorithm to integrate the processing results of different storage nodes to obtain the final analysis results;

[0097] Based on the final analysis results, feedback is provided to optimize the processing flow and improve overall processing efficiency and quality.

[0098] An electronic device, comprising:

[0099] A memory storing executable program code;

[0100] a processor coupled to the memory;

[0101] The processor calls the executable program code stored in the memory to execute the image and data analysis and processing method for ultra-large files as described above.

[0102] A computer storage medium stores computer instructions, which, when called, are used to execute the above-mentioned image and data analysis and processing method for very large files.

[0103] The same or similar reference numerals correspond to the same or similar components;

[0104] The terms used in the drawings to describe positional relationships are only used for illustrative purposes and should not be construed as limiting this patent;

[0105] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not limitations on the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made on the basis of the above description. It is not necessary and impossible to list all the implementation methods here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A method for analyzing and processing images and data of very large files, characterized in that: Using intelligent segmentation algorithm, the super large file is automatically divided into multiple sub-files, and an index is created for each sub-file; Build a distributed storage network based on edge computing and cloud computing, including edge storage nodes and cloud storage nodes; According to the file index, sub-files are allocated to different storage nodes, and an adaptive load balancing algorithm is used to dynamically schedule computing resources to achieve parallel processing; On each of the storage nodes, the sub-files are processed using GPU-accelerated image processing and data analysis algorithms, and deep learning technology is used to intelligently analyze the processing results; The processing results of the storage nodes are aggregated, the final analysis results are obtained through data fusion technology, and the processing flow is optimized according to the analysis results.

2. The image and data analysis and processing method for super-large files according to claim 1, characterized in that: The method of using an intelligent segmentation algorithm to automatically segment a large file into multiple sub-files and creating an index for each sub-file includes the following specific steps: Analyze the structure and content of the super-large file to determine a suitable splitting reference point; According to a preset segmentation rule, the super large file is segmented into a plurality of sub-files; Unique index information is generated for each sub-file, and the index information at least includes a starting position, an ending position, a data type, and associated metadata of the sub-file.

3. The image and data analysis and processing method for super large files according to claim 1, characterized in that: The specific steps of constructing a distributed storage network based on edge computing and cloud computing include: Selecting geographically dispersed edge storage nodes to ensure low latency and high availability of data processing; Deploy the cloud storage nodes with powerful computing and storage capabilities to support large-scale data processing needs; A communication mechanism is established between the edge storage node and the cloud storage node to ensure data synchronization and task coordination.

4. The image and data analysis and processing method for super-large files according to claim 1, characterized in that: The specific steps of allocating sub-files to different storage nodes according to the file index and using an adaptive load balancing algorithm to dynamically schedule computing resources to achieve parallel processing include: According to the index information of the sub-file, real-time monitoring of the load conditions of the edge storage node and the cloud storage node; Applying an adaptive load balancing algorithm to dynamically distribute the sub-files to the storage nodes with lighter loads; The computing resources are scheduled to ensure that the storage nodes can process the allocated sub-files in parallel.

5. The image and data analysis and processing method for super-large files according to claim 1, characterized in that: The image processing and data analysis algorithms accelerated by GPU include: Deploy a GPU-accelerated image processing unit on each of the storage nodes to increase the image processing speed; Using the parallel processing capability of GPU to efficiently process the image data in the sub-file; A data analysis algorithm is implemented to perform in-depth analysis on the data in the subfiles.

6. The image and data analysis and processing method for super-large files according to claim 1, characterized in that: The use of deep learning technology to intelligently analyze the processing results includes: Train deep learning models for intelligent analysis of image processing and data analysis results; Deploy a deep learning model on each of the storage nodes to implement real-time analysis of the sub-file processing results; Adjust the parameters of the deep learning model based on the analysis results to optimize the analysis accuracy.

7. The image and data analysis and processing method for super-large files according to claim 1, characterized in that: The specific steps of aggregating the processing results of the storage nodes and obtaining the final analysis results through data fusion technology include: Collect the processing results of the storage nodes to ensure the integrity and consistency of the data; Apply data fusion algorithm to integrate the processing results of different storage nodes to obtain the final analysis results; Based on the final analysis results, feedback is provided to optimize the processing flow and improve overall processing efficiency and quality.

8. An electronic device, characterized in that: The electronic device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the image and data analysis and processing method for ultra-large files as described in any one of claims 1-7.

9. A computer storage medium, characterized in that The computer storage medium stores computer instructions, which, when called, are used to execute the image and data analysis and processing method for super-large files as described in any one of claims 1-7.

Images