High-performance parallel file processing method and system in machine learning scenarios
By acquiring the number of images in real time and generating processing backup instructions in machine learning scenarios, creating parallel processing processes based on image features and adjusting the clustering and fitting of layers, solving a large number of problems of low image processing efficiency and achieving efficient images deal with.
Patent Information
- Application Number
- CN202411246775.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-06
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-09-06
AI Technical Summary
In machine learning scenarios, when faced with a large number of images, the prior art is difficult to efficiently process, resulting in too long processing cycles.
By obtaining the number of images to be processed in real time, when the preset threshold is reached, processing backup instructions are generated, and the clustering and fitting of image features and adjustment layers are created, and the image is used to use the fitted adjustment layer to perform the image. Processing, simplify the processing process and improve efficiency.
This greatly improves the image processing efficiency and simplifies the processing process. Although it reduces a certain processing effect, it significantly shortens the processing time.
Smart Images

Figure CN118760532B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of parallel image processing, and in particular to a high-performance parallel file processing method and system in a machine learning scenario. Background Art
[0002] Image processing and machine learning are two broad fields that have developed rapidly in recent years and have been widely used in various fields. Image processing focuses on processing, analyzing and understanding images, while machine learning focuses on modeling computers so that they can learn and make decisions autonomously from data. In image processing, machine learning techniques can be used for tasks such as image classification, detection, and segmentation, while in machine learning, image processing techniques can be used for data preprocessing, feature extraction, and other aspects.
[0003] In the prior art, when the model obtained by machine learning is used to process images, the processing time of a single image is actually not short. It just seems convenient that by inputting an image, a rendered image can be obtained. In fact, the computer is under great workload. When there are many images, the processing cycle is very long. How to improve the image processing efficiency when facing more images is the technical problem that the technical solution of the present invention aims to solve. Summary of the invention
[0004] The purpose of the present invention is to provide a high-performance parallel file processing method and system in a machine learning scenario to solve the problems raised in the above background technology.
[0005] To achieve the above object, the present invention provides the following technical solutions:
[0006] A high-performance parallel file processing method in a machine learning scenario, the method comprising:
[0007] Acquire the number of images to be processed in real time, and generate a processing backup instruction when the number of images reaches a preset number threshold;
[0008] Obtaining input images and output images of a machine learning model in real time based on processing backup instructions, and treating images with the same name as image pairs;
[0009] For each image pair, extracting image features from an input image, obtaining an output image and at least one adjustment layer of the input image, and establishing samples of the image features and the adjustment layers;
[0010] Cluster the samples based on image features. During the clustering process, count the image features and fit the adjustment layer.
[0011] Create parallel processing processes in which the images are processed based on the fitted adjustment layers.
[0012] As a further solution of the present invention: for each image pair, extracting image features from the input image, obtaining an output image and at least one adjustment layer of the input image, and establishing samples of the image features and the adjustment layer include:
[0013] Extract the layers of the input image under different channels, calculate the pixel difference of each layer, and select the layer with the smallest pixel difference as the feature layer;
[0014] Converting the feature layer into a co-occurrence matrix, and extracting image features based on the co-occurrence matrix;
[0015] Obtaining an output image and an input image, calculating a difference between the output image and the input image, determining an adjustment direction based on the difference, and randomly combining a preset number of adjustment layers based on the adjustment direction; the adjustment layers include a hue layer, a contrast layer, a brightness layer, and a saturation layer;
[0016] The randomly combined adjustment layers are superimposed on the input image to obtain a processed image;
[0017] Compare the differences between the processed image and the output image, and select the adjustment layer corresponding to the processed image with the smallest difference as the final adjustment layer;
[0018] Samples for building image features and final adjustment layers.
[0019] As a further solution of the present invention: the process of converting the feature layer into a co-occurrence matrix and extracting image features based on the co-occurrence matrix includes:
[0020] ;
[0021] In the formula, is the midpoint of the co-occurrence matrix The value of is the preset distance, , is the preset angle; The values include 0, , and ; For coordinate pairs The number of Indicates the line, Represents a column; Indicates the line difference, Indicates column difference;
[0022] Image features include:
[0023] Entropy: ; In the formula, is the entropy value, and is the number of rows and columns of the co-occurrence matrix, is the midpoint of the determined co-occurrence matrix The value of is A simplified representation of ;
[0024] energy: ; In the formula, For energy, and is the number of rows and columns of the co-occurrence matrix;
[0025] Uniformity: ; In the formula, is the uniformity, and is the number of rows and columns of the co-occurrence matrix.
[0026] As a further solution of the present invention: the process of comparing the difference between the processed image and the output image includes:
[0027] Obtain the histogram of the processed image at different scales, and obtain the histogram of the output image at different scales;
[0028] Taking the histogram of the output image as the target, the histogram of the processed image is scaled to the histogram of the output image based on the histogram matching technique, and the scaling ratio is recorded as the difference.
[0029] As a further solution of the present invention: the steps of clustering samples based on image features, performing statistics on image features during the clustering process, and fitting the adjustment layer include:
[0030] Compare image features, calculate distances between image features, and cluster samples according to the distances;
[0031] In the clustering process, the image features are counted to obtain an image feature set; the image features in the image feature set are in a parallel relationship;
[0032] The adjustment layer corresponding to each image feature counted is read, similar layers in the adjustment layer are superimposed, and the mean of each point in the layer is synchronously calculated to obtain the mean layer as the adjustment layer for this type of image feature.
[0033] As a further solution of the present invention: the step of creating a parallel processing process, in which the step of processing the image based on the fitted adjustment layer comprises:
[0034] Creating a parallel processing process, and receiving an image to be processed based on the parallel processing process;
[0035] Extracting image features of the to-be-processed images received by the parallel processing processes based on the same image feature extraction process;
[0036] Calculate the distance between the image feature and each type of image feature counted, and when the distance is less than a preset distance threshold, increase the number of matches of the corresponding type of image feature by one;
[0037] The matching times of each clustering result are queried, and the fitted adjustment layer in the clustering result corresponding to the maximum matching times is selected to process the image to be processed.
[0038] The technical solution of the present invention also provides a high-performance parallel file processing system in a machine learning scenario, the system comprising:
[0039] A quantity determination module is used to obtain the number of images to be processed in real time, and generate a processing backup instruction when the number of images reaches a preset quantity threshold;
[0040] An image pair building module, for obtaining input images and output images of a machine learning model in real time based on a processing backup instruction, and treating images with the same name as image pairs;
[0041] A sample generation module, for each image pair, extracts image features from an input image, obtains an output image and at least one adjustment layer of the input image, and creates samples of the image features and the adjustment layers;
[0042] The sample clustering module is used to cluster samples based on image features. During the clustering process, the image features are counted and the adjustment layer is fitted.
[0043] The parallel processing module is used to create a parallel processing process, in which the image is processed based on the fitted adjustment layer.
[0044] As a further solution of the present invention: the sample generation module includes:
[0045] The layer selection unit is used to extract the layers of the input image under different channels, calculate the pixel difference of each layer, and select the layer with the smallest pixel difference as the feature layer;
[0046] A first feature extraction unit, configured to convert a feature layer into a co-occurrence matrix, and extract image features based on the co-occurrence matrix;
[0047] A layer combination unit, used to obtain an output image and an input image, calculate a difference between the output image and the input image, determine an adjustment direction based on the difference, and randomly combine a preset number of adjustment layers based on the adjustment direction; the adjustment layers include a hue layer, a contrast layer, a brightness layer, and a saturation layer;
[0048] A layer overlay unit, used for overlaying the randomly combined adjustment layers onto the input image to obtain a processed image;
[0049] A difference comparison unit is used to compare the difference between the processed image and the output image, and select the adjustment layer corresponding to the processed image with the smallest difference as the final adjustment layer;
[0050] Create execution units for building samples of image features and final adjustment layers.
[0051] As a further solution of the present invention: the sample clustering module includes:
[0052] A distance calculation unit, used to compare image features, calculate the distance of image features, and cluster samples according to the distance;
[0053] A feature statistics unit is used to count image features in the clustering process to obtain an image feature set; the image features in the image feature set are in a parallel relationship;
[0054] The mean feature calculation unit is used to read the adjustment layer corresponding to each image feature counted, superimpose the same type of layers in the adjustment layer, and synchronously calculate the mean of each point in the layer to obtain the mean layer as the adjustment layer of this type of image feature.
[0055] As a further solution of the present invention: the parallel processing module includes:
[0056] A process generation unit, used for creating a parallel processing process and receiving an image to be processed based on the parallel processing process;
[0057] A second feature extraction unit is used to extract image features of the to-be-processed image received by the parallel processing process based on the same image feature extraction process;
[0058] A distance calculation and comparison unit is used to calculate the distance between the image feature and the various types of image features that have been counted. When the distance is less than a preset distance threshold, the number of matches of the corresponding type of image features is increased by one.
[0059] The execution unit is used to query the matching times of each type of clustering result, select the fitted adjustment layer in the clustering result corresponding to the maximum matching times, and process the image to be processed.
[0060] Compared with the prior art, the beneficial effects of the present invention are: when facing more image processing tasks, the present invention determines the adjustment layer according to the processed images before and after processing, and then creates a parallel processing process. The parallel processing process processes a part of the image based on the adjusted image. The processing method is very simple, and the adjustment layer can be superimposed on the image. The original complex processing process is converted into a simple image superposition process, which greatly improves the efficiency while only reducing a part of the processing effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention.
[0062] Figure 1 This is the overall flow chart of the high-performance parallel file processing method in the machine learning scenario.
[0063] Figure 2 This is a structural block diagram of a high-performance parallel file processing system in a machine learning scenario. DETAILED DESCRIPTION
[0064] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0065] Figure 1 The overall flow chart of the high-performance parallel file processing method in the machine learning scenario is as follows. In an embodiment of the present invention, a high-performance parallel file processing method in the machine learning scenario includes:
[0066] Step S100: acquiring the number of images to be processed in real time, and generating a processing backup instruction when the number of images reaches a preset number threshold;
[0067] Step S200: acquiring input images and output images of the machine learning model in real time based on the processing backup instruction, and treating images with the same name as image pairs;
[0068] Step S300: for each image pair, extract image features from the input image, obtain an output image and at least one adjustment layer of the input image, and establish samples of the image features and the adjustment layers;
[0069] Step S400: clustering samples based on image features, performing statistics on image features and fitting adjustment layers during the clustering process;
[0070] Step S500: creating a parallel processing process, in which the image is processed based on the fitted adjustment layer.
[0071] Common application areas of machine learning include image processing, such as image restoration and enhancement. A model is trained through machine learning, and the trained model can be used to process images. Processing images through the model requires high performance and is not very fast. When the number of images is large, the efficiency is not high.
[0072] In the technical solution of the present invention, the number of images to be processed is acquired in real time. When the number of images is large enough, the images are processed by another simplified process to improve efficiency, that is, a parallel processing process.
[0073] The technical solution provided by step S100 to step S500 can effectively reduce the processing time. The principle is that when the model obtained by deep learning processes the image, the processing process is recorded in real time, and the adjustment layer is determined according to the images before and after the processing. When a similar image is received, the adjustment layer is superimposed on the image, and the original complex processing process is converted into a simple image superposition process. Although the processing effect is reduced to a certain extent, the processing efficiency is greatly improved.
[0074] Specifically, the number of images to be processed is obtained in real time. When the number of images is large enough, the input image and output image of the model are obtained, and the input image and the output image are paired according to the image name to obtain an image pair; the image pair is the data before and after a processing process.
[0075] Feature extraction is performed on the input image to obtain image features, and then the difference between the output image and the input image is calculated to determine how many times the input image can be adjusted to almost reach the state of the output image, which is called an adjustment layer. The image features and the adjustment layer are connected as a sample.
[0076] The samples are clustered based on image features. After clustering, a set of image features and the comprehensive adjustment layer corresponding to the set are obtained. In the set, the image features are in a parallel relationship and there are multiple ones, but the comprehensive adjustment layer is single, and one type of image features corresponds to only one comprehensive adjustment layer.
[0077] When the number of images is large enough, a parallel processing process is created, and a part of the images are processed by the parallel processing process. For this part of the images, the same technical solution as step S300 is used to extract image features, match them with the clustering results, determine which category of images the new image belongs to, read the comprehensive adjustment layer of this type of image, and superimpose the images to simulate the processing effect.
[0078] In an example of the technical solution of the present invention, the steps of extracting image features from the input image, obtaining the output image and at least one adjustment layer of the input image, and establishing samples of the image features and the adjustment layer include:
[0079] Extract the layers of the input image under different channels, calculate the pixel difference of each layer, and select the layer with the smallest pixel difference as the feature layer;
[0080] Converting the feature layer into a co-occurrence matrix, and extracting image features based on the co-occurrence matrix;
[0081] Obtaining an output image and an input image, calculating a difference between the output image and the input image, determining an adjustment direction based on the difference, and randomly combining a preset number of adjustment layers based on the adjustment direction; the adjustment layers include a hue layer, a contrast layer, a brightness layer, and a saturation layer;
[0082] The randomly combined adjustment layers are superimposed on the input image to obtain a processed image;
[0083] Compare the differences between the processed image and the output image, and select the adjustment layer corresponding to the processed image with the smallest difference as the final adjustment layer;
[0084] Samples for building image features and final adjustment layers.
[0085] The above content specifically describes step S200 , which focuses on two parts: one is the image feature extraction process, and the other is the adjustment layer generation process.
[0086] The image feature extraction process is as follows:
[0087] Extract the layers of the input image under different channels. An image usually has three channels. Calculate the pixel difference of each layer and select the layer with the smallest pixel difference as the feature layer. This step is equivalent to extracting a relatively important channel in the image, called the feature layer. Convert the feature layer into a co-occurrence matrix and extract the image features based on the co-occurrence matrix.
[0088] The generation process of the adjustment layer is as follows:
[0089] An output image and an input image are obtained, the difference between the output image and the input image is calculated, and an adjustment direction is determined based on the difference. The adjustment direction is used to represent the change trend of the input image and the output image, and can be obtained by subtracting the input image from the output image at different scales. The scale includes hue, contrast, brightness, saturation, etc. (existing image adjustment scales can be used as a reference).
[0090] At different scales, adjustment layers are randomly created and combined based on the adjustment direction, the input image is adjusted, and then the difference between the adjusted image and the output image is calculated. When the difference is small enough, the combined adjustment layer is retained as the final adjustment layer.
[0091] The above content is essentially an enumeration test scheme. Under the image change trend (the transformation trend from input image to output image), multiple adjustment schemes are randomly created, and the best scheme is selected from multiple adjustment schemes.
[0092] In the above content, the process of converting the feature layer into a co-occurrence matrix and extracting image features based on the co-occurrence matrix includes:
[0093] ;
[0094] In the formula, is the midpoint of the co-occurrence matrix The value of is the preset distance, , is the preset angle; The values include 0, , and ; For coordinate pairs The number of Indicates the line, Represents a column; Indicates the line difference, Indicates column difference;
[0095] Image features include:
[0096] Entropy: ; In the formula, is the entropy value, and is the number of rows and columns of the co-occurrence matrix, is the midpoint of the determined co-occurrence matrix The value of is A simplified representation of ;
[0097] energy: ; In the formula, For energy, and is the number of rows and columns of the co-occurrence matrix;
[0098] Uniformity: ; In the formula, is the uniformity, and is the number of rows and columns of the co-occurrence matrix.
[0099] The above content provides a specific co-occurrence matrix generation scheme and an eigenvalue calculation scheme based on the co-occurrence matrix. The meaning of the co-occurrence matrix is:
[0100] In the image, in the preset In the direction, if two adjacent pixels (distance is ) pixel value, one is , and the other is , then we take it as the coordinate pair that meets the conditions, count all the coordinate pairs that meet the conditions in the image, and take their number as the midpoint of the co-occurrence matrix The value of .
[0101] Based on this, the frequency of each value in the co-occurrence matrix is calculated, and then the entropy, energy and uniformity are calculated based on the frequency as image features, thereby achieving the feature extraction task; the entropy reflects the amount of information, the energy reflects the uniformity and smoothness, and the uniformity reflects the uniformity of the image.
[0102] Furthermore, the process of comparing the difference between the processed image and the output image includes:
[0103] Obtain the histogram of the processed image at different scales, and obtain the histogram of the output image at different scales;
[0104] Taking the histogram of the output image as the target, the histogram of the processed image is scaled to the histogram of the output image based on the histogram matching technique, and the scaling ratio is recorded as the difference.
[0105] The above content provides a simplest difference calculation process, which converts the histogram of the processed image into the histogram of the output image based on the histogram matching technology. This process only requires scaling, and does not require complete matching. The horizontal axis can be the same. This process can be compared to the curve tool in PS.
[0106] In a preferred embodiment of the technical solution of the present invention, the steps of clustering samples based on image features, performing statistics on image features during the clustering process, and fitting the adjustment layer include:
[0107] Compare image features, calculate distances between image features, and cluster samples according to the distances;
[0108] In the clustering process, the image features are counted to obtain an image feature set; the image features in the image feature set are in a parallel relationship;
[0109] The adjustment layer corresponding to each image feature counted is read, similar layers in the adjustment layer are superimposed, and the mean of each point in the layer is synchronously calculated to obtain the mean layer as the adjustment layer for this type of image feature.
[0110] The above content provides a specific image clustering process, comparing image features and calculating the distance between image features. The image feature is an array, and the distance between arrays can be calculated using Euclidean distance or cosine similarity. After the distance is calculated, the existing clustering algorithm can be used to cluster the samples.
[0111] In the clustering process, all image features are counted to obtain an image feature set, and the image features in the image feature set are in a parallel relationship; however, for the adjustment layer, similar layers in the adjustment layer are superimposed, and then the mean of each pixel point in the superposition process is calculated. The obtained layer is called the mean layer, and the mean layer is used as the adjustment layer for this type of image.
[0112] In some examples of the technical solution of the present invention, the step of creating a parallel processing process, in which the step of processing the image based on the fitted adjustment layer comprises:
[0113] Creating a parallel processing process, and receiving an image to be processed based on the parallel processing process;
[0114] Extracting image features of the to-be-processed images received by the parallel processing processes based on the same image feature extraction process;
[0115] Calculate the distance between the image feature and each type of image feature counted, and when the distance is less than a preset distance threshold, increase the number of matches of the corresponding type of image feature by one;
[0116] The matching times of each clustering result are queried, and the fitted adjustment layer in the clustering result corresponding to the maximum matching times is selected to process the image to be processed.
[0117] The above content is a specific application process. When there are enough images, a parallel processing process is created. Based on the parallel processing process, a part of the image processing task is shared, and features are extracted from the shared images to obtain image features of the new images. Then, the distance between the new images and the image features of all backups is calculated to query similar images. The number of similar images in each category of images is obtained. The larger the number, the more consistent the new image is with the image of that category. The adjustment layer of the image of that category can be read as the final layer.
[0118] It is worth mentioning that the image feature extraction process and the distance calculation process of the new image both use the above content to provide the image feature extraction process (image feature extraction stage) and the distance calculation process (image feature clustering stage).
[0119] Figure 2: is a structural block diagram of a high-performance parallel file processing system in a machine learning scenario. In a preferred embodiment of the technical solution of the present invention, a high-performance parallel file processing system in a machine learning scenario is also provided. The high-performance parallel file processing system 10 in the machine learning scenario includes:
[0120] The quantity determination module 11 is used to obtain the number of images to be processed in real time, and generate a processing backup instruction when the number of images reaches a preset quantity threshold;
[0121] An image pair construction module 12, for acquiring input images and output images of a machine learning model in real time based on the processing backup instruction, and treating images with the same name as image pairs;
[0122] The sample generation module 13 extracts image features from the input image for each image pair, obtains an output image and at least one adjustment layer of the input image, and creates samples of the image features and the adjustment layers;
[0123] The sample clustering module 14 is used to cluster the samples based on the image features. During the clustering process, the image features are counted and the adjustment layer is fitted;
[0124] The parallel processing module 15 is used to create a parallel processing process, in which the image is processed based on the fitted adjustment layer.
[0125] Furthermore, the sample generation module 13 includes:
[0126] The layer selection unit is used to extract the layers of the input image under different channels, calculate the pixel difference of each layer, and select the layer with the smallest pixel difference as the feature layer;
[0127] A first feature extraction unit, configured to convert a feature layer into a co-occurrence matrix, and extract image features based on the co-occurrence matrix;
[0128] A layer combination unit, used to obtain an output image and an input image, calculate a difference between the output image and the input image, determine an adjustment direction based on the difference, and randomly combine a preset number of adjustment layers based on the adjustment direction; the adjustment layers include a hue layer, a contrast layer, a brightness layer, and a saturation layer;
[0129] A layer overlay unit, used for overlaying the randomly combined adjustment layers onto the input image to obtain a processed image;
[0130] A difference comparison unit is used to compare the difference between the processed image and the output image, and select the adjustment layer corresponding to the processed image with the smallest difference as the final adjustment layer;
[0131] Create execution units for building samples of image features and final adjustment layers.
[0132] Specifically, the sample clustering module 14 includes:
[0133] A distance calculation unit, used to compare image features, calculate the distance of image features, and cluster samples according to the distance;
[0134] A feature statistics unit is used to count image features in the clustering process to obtain an image feature set; the image features in the image feature set are in a parallel relationship;
[0135] The mean feature calculation unit is used to read the adjustment layer corresponding to each image feature counted, superimpose the same type of layers in the adjustment layer, and synchronously calculate the mean of each point in the layer to obtain the mean layer as the adjustment layer of this type of image feature.
[0136] Furthermore, the parallel processing module 15 includes:
[0137] A process generation unit, used for creating a parallel processing process and receiving an image to be processed based on the parallel processing process;
[0138] A second feature extraction unit is used to extract image features of the to-be-processed image received by the parallel processing process based on the same image feature extraction process;
[0139] A distance calculation and comparison unit is used to calculate the distance between the image feature and the various types of image features that have been counted. When the distance is less than a preset distance threshold, the number of matches of the corresponding type of image features is increased by one.
[0140] The execution unit is used to query the matching times of each type of clustering result, select the fitted adjustment layer in the clustering result corresponding to the maximum matching times, and process the image to be processed.
[0141] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A high-performance parallel file processing method in a machine learning scenario, characterized in that: The method comprises: Acquire the number of images to be processed in real time, and generate a processing backup instruction when the number of images reaches a preset number threshold; Obtaining input images and output images of a machine learning model in real time based on processing backup instructions, and treating images with the same name as image pairs; For each image pair, extracting image features from an input image, obtaining an output image and at least one adjustment layer of the input image, and establishing samples of the image features and the adjustment layers; Cluster the samples based on image features. During the clustering process, count the image features and fit the adjustment layer. Creating a parallel processing process, in which the image is processed based on the fitted adjustment layer; The steps of extracting image features from the input image for each image pair, obtaining an output image and at least one adjustment layer of the input image, and establishing samples of the image features and the adjustment layers include: Extract the layers of the input image under different channels, calculate the pixel difference of each layer, and select the layer with the smallest pixel difference as the feature layer; Converting the feature layer into a co-occurrence matrix, and extracting image features based on the co-occurrence matrix; Obtaining an output image and an input image, calculating a difference between the output image and the input image, determining an adjustment direction based on the difference, and randomly combining a preset number of adjustment layers based on the adjustment direction; the adjustment layers include a hue layer, a contrast layer, a brightness layer, and a saturation layer; The randomly combined adjustment layers are superimposed on the input image to obtain a processed image; Compare the differences between the processed image and the output image, and select the adjustment layer corresponding to the processed image with the smallest difference as the final adjustment layer; Build samples of image features and final adjustment layers; The process of converting the feature layer into a co-occurrence matrix and extracting image features based on the co-occurrence matrix includes: ; In the formula, is the midpoint of the co-occurrence matrix The value of is the preset distance, , is the preset angle; The values include 0, , and ; For coordinate pairs The number of Indicates the line, Represents a column; Indicates the line difference, Indicates column difference; Image features include: Entropy: ; In the formula, is the entropy value, and is the number of rows and columns of the co-occurrence matrix, is the midpoint of the determined co-occurrence matrix The value of A simplified representation of ; energy: ; In the formula, for energy; Uniformity: ; In the formula, For uniformity.
2. The high-performance parallel file processing method in a machine learning scenario according to claim 1, characterized in that: The process of comparing the differences between the processed image and the output image includes: Obtain the histogram of the processed image at different scales, and obtain the histogram of the output image at different scales; Taking the histogram of the output image as the target, the histogram of the processed image is scaled to the histogram of the output image based on the histogram matching technique, and the scaling ratio is recorded as the difference.
3. The high-performance parallel file processing method in a machine learning scenario according to claim 1, characterized in that: The steps of clustering samples based on image features, performing statistics on image features during the clustering process, and fitting the adjustment layer include: Compare image features, calculate distances between image features, and cluster samples according to the distances; In the clustering process, the image features are counted to obtain an image feature set; the image features in the image feature set are in a parallel relationship; The adjustment layer corresponding to each image feature counted is read, similar layers in the adjustment layer are superimposed, and the mean of each point in the layer is synchronously calculated to obtain the mean layer as the adjustment layer for this type of image feature.
4. The high-performance parallel file processing method in a machine learning scenario according to claim 3, characterized in that: The step of creating a parallel processing process, in which the image is processed based on the fitted adjustment layer, includes: Creating a parallel processing process, and receiving an image to be processed based on the parallel processing process; Extracting image features of the to-be-processed images received by the parallel processing processes based on the same image feature extraction process; Calculate the distance between the image feature and each type of image feature counted, and when the distance is less than a preset distance threshold, increase the number of matches of the corresponding type of image feature by one; The matching times of each clustering result are queried, and the fitted adjustment layer in the clustering result corresponding to the maximum matching times is selected to process the image to be processed.
5. A high-performance parallel file processing system in a machine learning scenario, characterized in that: The system comprises: A quantity determination module is used to obtain the number of images to be processed in real time, and generate a processing backup instruction when the number of images reaches a preset quantity threshold; An image pair building module, for obtaining input images and output images of a machine learning model in real time based on a processing backup instruction, and treating images with the same name as image pairs; A sample generation module, for each image pair, extracts image features from an input image, obtains an output image and at least one adjustment layer of the input image, and creates samples of the image features and the adjustment layers; The sample clustering module is used to cluster samples based on image features. During the clustering process, the image features are counted and the adjustment layer is fitted. A parallel processing module, used to create a parallel processing process, in which the image is processed based on the fitted adjustment layer; The sample generation module comprises: The layer selection unit is used to extract the layers of the input image under different channels, calculate the pixel difference of each layer, and select the layer with the smallest pixel difference as the feature layer; A first feature extraction unit, configured to convert a feature layer into a co-occurrence matrix, and extract image features based on the co-occurrence matrix; A layer combination unit, used to obtain an output image and an input image, calculate a difference between the output image and the input image, determine an adjustment direction based on the difference, and randomly combine a preset number of adjustment layers based on the adjustment direction; the adjustment layers include a hue layer, a contrast layer, a brightness layer, and a saturation layer; A layer overlay unit, used for overlaying the randomly combined adjustment layers onto the input image to obtain a processed image; A difference comparison unit is used to compare the difference between the processed image and the output image, and select the adjustment layer corresponding to the processed image with the smallest difference as the final adjustment layer; Establishing execution units for establishing samples of image features and final adjustment layers; The process of converting the feature layer into a co-occurrence matrix and extracting image features based on the co-occurrence matrix includes: ; In the formula, is the midpoint of the co-occurrence matrix The value of is the preset distance, , is the preset angle; The values include 0, , and ; For coordinate pairs The number of Indicates the line, Represents a column; Indicates the line difference, Indicates column difference; Image features include: Entropy: ; In the formula, is the entropy value, and is the number of rows and columns of the co-occurrence matrix, is the midpoint of the determined co-occurrence matrix The value of A simplified representation of ; energy: ; In the formula, for energy; Uniformity: ; In the formula, For uniformity.
6. The high-performance parallel file processing system in a machine learning scenario according to claim 5, characterized in that: The sample clustering module comprises: A distance calculation unit, used to compare image features, calculate the distance of image features, and cluster samples according to the distance; A feature statistics unit is used to count image features in the clustering process to obtain an image feature set; the image features in the image feature set are in a parallel relationship; The mean feature calculation unit is used to read the adjustment layer corresponding to each image feature counted, superimpose the same type of layers in the adjustment layer, and synchronously calculate the mean of each point in the layer to obtain the mean layer as the adjustment layer of this type of image feature.
7. The high-performance parallel file processing system in a machine learning scenario according to claim 6, characterized in that: The parallel processing module comprises: A process generation unit, used for creating a parallel processing process and receiving an image to be processed based on the parallel processing process; A second feature extraction unit is used to extract image features of the to-be-processed image received by the parallel processing process based on the same image feature extraction process; A distance calculation and comparison unit is used to calculate the distance between the image feature and the various types of image features that have been counted. When the distance is less than a preset distance threshold, the number of matches of the corresponding type of image features is increased by one. The execution unit is used to query the matching times of each type of clustering result, select the fitted adjustment layer in the clustering result corresponding to the maximum matching times, and process the image to be processed.
