Image data anomaly analysis system based on neural network

Abstract

The invention relates to the technical field of image data analysis, and discloses an image data anomaly analysis system based on a neural network. The analysis system is composed of a data preprocessing module, a data storage and management module, a data processing module, a data consistency and coordination module, a fault tolerance and recovery module, a performance optimization module and a data transmission and scheduling module. By standardizing the image data and performing minimum-maximum standardization, the consistency and uniformity of the data are ensured, the data are fragmented and stored in the distributed file system, the problem of single-point failure is avoided, the reliability and recovery capability of the data are improved, the image data are processed in parallel through the distributed computing framework, and the image processing efficiency is improved. The overall processing time is reduced, and the beneficial effect that the data processing result is not abnormal when the large-scale image data is processed is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of image data analysis, and in particular to an image data anomaly analysis system based on a neural network. Background Art

[0002] Scene images refer to coherent images composed of a background with a reasonable spatial distribution and some scattered objects in a real environment. As a carrier of rich semantic information, their number is growing explosively. Due to the structural complexity of image data itself, the diversity of information contained, and the multidimensionality of time and space, effective organization and management of large-scale image data and rapid retrieval of images required by users have become hot topics. The problems with traditional image feature extraction and retrieval methods are that the recommended image data based on the results of image data analysis are largely inconsistent with people's expectations, that is, the semantics of the recommended image data analysis are quite different from the actual search image semantics, resulting in a low overall recommendation efficiency.

[0003] In order to solve the defects of the existing technology, a BP network-based image data analysis system with publication number CN 115758166 A was found through search. The system uses an iterative algorithm to fuse the prediction results of multiple BP neural networks to form a strong prediction learner to predict the emotional semantics of scene images: it uses distributed processing of a multi-node architecture to allow multiple computers on the network to interconnect and collaborate to complete retrieval calculations and storage of large-scale image data; and outputs the retrieval results.

[0004] Although the aforementioned technical solutions solve the problem of improving the recommendation rate through image data analysis, when processing large-scale image data in a distributed system, network delays, node failures or inconsistencies may lead to inconsistencies in data processing results, affecting the overall retrieval efficiency. The data processed by a certain node may contain anomalies, which will affect the performance and data consistency of the entire system. Summary of the invention

[0005] Technical issues solved:

[0006] In view of the shortcomings of the prior art, the present invention provides an image data anomaly analysis system based on a neural network, which has the advantages of preventing anomalies in data processing results when processing large-scale image data, avoiding inconsistencies and abnormal data, and solving the problems of the above-mentioned technology.

[0007] Technical solution:

[0008] To achieve the above object, the present invention provides the following technical solutions: an image data anomaly analysis system based on a neural network, the analysis system consisting of a data preprocessing module, a data storage and management module, a data processing module, a data consistency and coordination module, a fault tolerance and recovery module, a performance optimization module and a data transmission and scheduling module;

[0009] The data preprocessing module is used to perform standardization processing on the image data and unify the data format and range;

[0010] The data storage and management module is used to store the image data in pieces in a distributed file system;

[0011] The data processing module uses a distributed computing framework to process image data in parallel, dynamically allocates tasks according to node load, and optimizes resource usage;

[0012] The data consistency and coordination module uses the Paxos protocol to reach consensus through negotiation between multiple nodes to ensure that data updates of all nodes are consistent;

[0013] The fault tolerance and recovery module is used to monitor node status in real time, detect and respond to faults;

[0014] The performance optimization module uses data compression technology to reduce data transmission volume, improve network bandwidth utilization, and uses consistent hashing algorithm to optimize data and task allocation, thereby improving system scalability;

[0015] The data transmission and scheduling module uses a consistent hashing algorithm to optimize the allocation of data and tasks, reduce data skew, and improve the scalability of the system;

[0016] The parallel processing expression in the distributed computing in the data storage and management module is:

[0017] Suppose there are N computing nodes and T image data tasks, where each task ti is assigned to node j, and the processing time of node j is P j (t i ), the total processing time of a task is the sum of the processing time of all tasks on node j;

[0018]

[0019] Among them: tasks j represents the set of all tasks assigned to node j;

[0020] The total processing time of the global task is expressed as:

[0021]

[0022] Among them: nodes represents the set of all computing nodes.

[0023] Preferably, the data storage and management module further includes node load dynamic allocation, and the node load dynamic allocation expression is:

[0024] In dynamic load balancing, the distribution of tasks will be adjusted according to the current node load. Let the current load of each node j be L j , assign tasks T according to the load i To node j, load L j Defined according to the processing time and task volume of node j:

[0025]

[0026] Among them: New Task Processing Time is the processing time of the new task;

[0027] To optimize resource usage, tasks are distributed by minimizing node load:

[0028]

[0029] Where: t i is the task to be assigned; node j is the target node; L j is the load of node j.

[0030] Preferably, the purpose of the standardization process in the data preprocessing module is to adjust the data to a distribution with a mean of 0 and a standard deviation of 1. Assume that each pixel value x of the image data needs to be standardized, with a mean of μ and a standard deviation of σ:

[0031]

[0032] Where: x is the pixel value of the original image data; μ is the mean value of the image data; σ is the standard deviation of the image data; x′ is the standardized pixel value;

[0033] The data preprocessing module also includes minimum-maximum standardization, expressed as:

[0034] Minimum-maximum standardization scales the data to the specified range, [0,1], and sets the minimum value of the original data to x min , the maximum value is x max :

[0035]

[0036] Where: x is the pixel value of the original image data; x min is the minimum value of the image data; x maxis the maximum value of the image data; x′ is the normalized pixel value.

[0037] Preferably, the data preprocessing module unifies the data format by converting the data into a specific format: converting the image data from RGB to a grayscale image, assuming that the pixel value of the RGB image is (R, G, B), and converting to a grayscale value G using the following formula:

[0038] G=0.2989*R+0.5870*G+0.1140*B

[0039] Where: R is the pixel value of the red channel; G is the pixel value of the green channel; B is the pixel value of the blue channel; G is the pixel value of the grayscale image;

[0040] Before the standardization process, the mean and standard deviation need to be calculated. Suppose the image data has a total of N pixel values ​​x i :

[0041]

[0042] Where: μ is the mean value of the image data; is the sum of all pixel values; N is the total number of pixel values.

[0043] Preferably, the node load calculation expression in the data processing module is:

[0044] L j =P j / T j

[0045] Where: P j is the total processing time of the tasks processed on node j; T j is the number of tasks currently being processed on node j; L j is the load of node j.

[0046] Preferably, the load balancing target expression in the data processing module is:

[0047]

[0048] Where: L target is the ideal load level; Σ j∈nodes P j is the sum of processing time of all nodes; N is the number of nodes.

[0049] Preferably, the data processing module dynamically allocates tasks to achieve load balancing, and optimizes task allocation by minimizing the difference between each node load and the target load. The adjusted node load calculation formula is:

[0050] Reassign Task t i to node j such that|Lj-L target |is minimized

[0051] Where: t i is the task to be reallocated; node j is the target node for receiving task t i ; L j is the load of node j; L target is the ideal load level; |L j ―L target | is the absolute difference between the load of node j and the target load.

[0052] Preferably, the proposal number expression of the proposal in the data consistency and coordination module is:

[0053] pi>pj (for all proposals i that have accepted proposal number pj

[0054] The expression of the preparation stage is:

[0055]

[0056] p is the current proposal number; p accept is the maximum proposal number that node j has accepted; v accept is the proposal value corresponding to node j; A j is the set of accepted proposals of node j;

[0057] Acceptance stage: p>p accept

[0058] Where: p is the current proposal number; p accept is the maximum proposal number accepted by node j.

[0059] Preferably, the node health status monitoring expression in the fault tolerance and recovery module is:

[0060] S i =H i (t)

[0061] Where: S i is the health status of node i at time t; H i (t): Health status function of node i at time t.

[0062] Preferably, the data compression expression in the performance optimization module is:

[0063] CR=D original / D compressed

[0064] Where: CR is the compression ratio; D original is the original data size D compressed The compressed data size.

[0065] Compared with the prior art, the present invention provides an image data anomaly analysis system based on a neural network, which has the following beneficial effects:

[0066] 1. The present invention ensures the consistency and uniformity of data by standardizing image data and performing minimum-maximum standardization, stores data in a distributed file system in pieces, avoids the problem of single point failure, improves data reliability and recovery capability, processes image data in parallel through a distributed computing framework, reduces the overall processing time, improves the processing capability of the system, and thus avoids exceptions caused by long processing time. The Paxos protocol is used to ensure consistent data updates between all nodes, avoiding processing exceptions caused by inconsistent data. The data is compressed to reduce the transmission volume and improve bandwidth utilization, thereby reducing processing exceptions caused by data transmission delays, achieving the beneficial effect of no abnormalities in data processing results when processing large-scale image data.

[0067] 2. The present invention adopts the Paxos protocol for consistency maintenance to ensure that data updates between multiple nodes are negotiated and agreed upon. Only data updates agreed upon by the majority of nodes will take effect, thereby avoiding inconsistencies caused by data asynchrony between nodes, reducing transmission volume through data compression, and improving bandwidth utilization. At the same time, the consistent hashing algorithm is used to optimize the allocation of data and tasks, effectively reducing the risk of data loss or damage during transmission, and further maintaining data consistency. The consistent hashing algorithm is used to optimize the allocation of data and tasks, reduce data skew, and ensure that the load of each node is uniform. The optimization strategy avoids processing anomalies caused by uneven data distribution, ensures that each node can accurately process the tasks assigned to it, and achieves the beneficial effect of avoiding inconsistencies and abnormal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 Schematic diagram of the system of the present invention. DETAILED DESCRIPTION

[0069] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] See also Figure 1, an image data anomaly analysis system based on neural network, the analysis system consists of a data preprocessing module, a data storage and management module, a data processing module, a data consistency and coordination module, a fault tolerance and recovery module, a performance optimization module and a data transmission and scheduling module;

[0071] The data preprocessing module is used to perform standardization processing on the image data and unify the data format and range;

[0072] The data storage and management module is used to store the image data in pieces in a distributed file system;

[0073] The data processing module uses a distributed computing framework to process image data in parallel, dynamically allocates tasks according to node load, and optimizes resource usage;

[0074] The data consistency and coordination module uses the Paxos protocol to reach consensus through negotiation between multiple nodes to ensure that data updates of all nodes are consistent;

[0075] The fault tolerance and recovery module is used to monitor node status in real time, detect and respond to faults;

[0076] The performance optimization module uses data compression technology to reduce data transmission volume, improve network bandwidth utilization, and uses consistent hashing algorithm to optimize data and task allocation, thereby improving system scalability;

[0077] The data transmission and scheduling module uses a consistent hashing algorithm to optimize the allocation of data and tasks, reduce data skew, and improve the scalability of the system;

[0078] The parallel processing expression in the distributed computing in the data storage and management module is:

[0079] Suppose there are N computing nodes and T image data tasks, where each task ti is assigned to node j, and the processing time of node j is P j (t i ), the total processing time of a task is the sum of the processing time of all tasks on node j;

[0080]

[0081] in: represents the set of all tasks assigned to node j;

[0082] The total processing time of the global task is expressed as:

[0083]

[0084] Among them: nodes represents the set of all computing nodes.

[0085] Standardization: Ensure the consistency and uniformity of the data by standardizing the image data and performing minimum-maximum standardization. This helps reduce bias in the data, making it more suitable for model processing, thereby reducing the possibility of abnormal results.

[0086] Unified data format: Convert RGB images to grayscale images to unify the data format, reduce the processing complexity brought by different color spaces, and further reduce the occurrence of abnormal situations.

[0087] Sharded storage: Storing data in shards in a distributed file system avoids single point failure problems and improves data reliability and recovery capabilities.

[0088] Dynamic load distribution: Adjust task distribution according to the load of the node, optimize resource usage, and prevent processing exceptions due to overload of some nodes.

[0089] Parallel processing: By processing image data in parallel through a distributed computing framework, the overall processing time is reduced and the processing capacity of the system is improved, thus avoiding anomalies caused by long processing time.

[0090] Load balancing: Dynamically adjust task allocation to ensure that tasks are evenly distributed among nodes, preventing some nodes from being overloaded, thereby avoiding anomalies that may be caused by uneven load.

[0091] Consistency maintenance: The Paxos protocol is used to ensure consistent data updates among all nodes, avoiding processing exceptions caused by inconsistent data.

[0092] Health status monitoring: Real-time monitoring of the health status of nodes, timely detection and response to faults, ensuring system stability and reliability, and preventing data processing anomalies caused by faults.

[0093] Data compression: Reduce the amount of data transmitted by compressing it, improve bandwidth utilization, and thus reduce processing anomalies caused by data transmission delays.

[0094] Optimize data distribution: Use consistent hashing algorithms to optimize the distribution of data and tasks, reduce data skew, improve system scalability, ensure that tasks and data are evenly distributed among nodes, and avoid problems caused by uneven processing load.

[0095] Optimize data distribution: The consistent hashing algorithm is also used to optimize the distribution of data and tasks, reduce data skew, and improve the scalability and processing efficiency of the system.

[0096] Preferably, the data storage and management module further includes node load dynamic allocation, and the node load dynamic allocation expression is:

[0097] In dynamic load balancing, the distribution of tasks will be adjusted according to the current node load. Let the current load of each node j be L j , assign tasks T according to the load i To node j, load L j Defined according to the processing time and task volume of node j:

[0098]

[0099] Among them: New Task Processing Time is the processing time of the new task;

[0100] To optimize resource usage, tasks are distributed by minimizing node load:

[0101]

[0102] Where: t i is the task to be assigned; node j is the target node; L j is the load of node j.

[0103] Preferably, the purpose of the standardization process in the data preprocessing module is to adjust the data to a distribution with a mean of 0 and a standard deviation of 1. Assume that each pixel value x of the image data needs to be standardized, with a mean of μ and a standard deviation of σ:

[0104]

[0105] Where: x is the pixel value of the original image data; μ is the mean value of the image data; σ is the standard deviation of the image data; x′ is the standardized pixel value;

[0106] The data preprocessing module also includes minimum-maximum standardization, expressed as:

[0107] Minimum-maximum standardization scales the data to the specified range, [0,1], and sets the minimum value of the original data to x min , the maximum value is x max :

[0108]

[0109] Where: x is the pixel value of the original image data; x min is the minimum value of the image data; x max is the maximum value of the image data; x′ is the normalized pixel value.

[0110] Preferably, the data preprocessing module unifies the data format by converting the data into a specific format: converting the image data from RGB to a grayscale image, assuming that the pixel value of the RGB image is (R, G, B), and converting to a grayscale value G using the following formula:

[0111] G=0.2989*R+0.5870*G+0.1140*B

[0112] Where: R is the pixel value of the red channel; G is the pixel value of the green channel; B is the pixel value of the blue channel; G is the pixel value of the grayscale image;

[0113] Before the standardization process, the mean and standard deviation need to be calculated. Suppose the image data has a total of N pixel values ​​x i :

[0114]

[0115] Where: μ is the mean value of the image data; is the sum of all pixel values; N is the total number of pixel values.

[0116] Preferably, the node load calculation expression in the data processing module is:

[0117] L j =P j / T j

[0118] Where: P j is the total processing time of the tasks processed on node j; T j is the number of tasks currently being processed on node j; L j is the load of node j.

[0119] Preferably, the load balancing target expression in the data processing module is:

[0120]

[0121] Where: L target is the ideal load level; Σ j∈nodes P j is the sum of processing time of all nodes; N is the number of nodes.

[0122] Preferably, the data processing module dynamically allocates tasks to achieve load balancing, and optimizes task allocation by minimizing the difference between each node load and the target load. The adjusted node load calculation formula is:

[0123] Reassign Task t i to node j such that|L j―L target |is minmized

[0124] Where: t i is the task to be reallocated; node j is the target node for receiving task t i ; L j is the load of node j; L target is the ideal load level; |L j ―L target | is the absolute difference between the load of node j and the target load.

[0125] Preferably, the proposal number expression of the proposal in the data consistency and coordination module is:

[0126] p i > j (For all accepted proposals with number p j Proposal

[0127] The expression of the preparation stage is:

[0128]

[0129] p is the current proposal number; p accept is the maximum proposal number that node j has accepted; v accept is the proposal value corresponding to node j; A j is the set of accepted proposals of node j;

[0130] Acceptance stage: p>p accept

[0131] Where: p is the current proposal number; p accept is the maximum proposal number accepted by node j.

[0132] Preferably, the node health status monitoring expression in the fault tolerance and recovery module is:

[0133] S i =H i (t)

[0134] Where: S i is the health status of node i at time t; H i (t): Health status function of node i at time t.

[0135] Preferably, the data compression expression in the performance optimization module is:

[0136] CR=d original / D compressed

[0137] Where: CR is the compression ratio; Doriginal is the original data size D compressed The compressed data size.

[0138] Parallel processing and load balancing: Data is processed in parallel through a distributed computing framework, and tasks are dynamically assigned to ensure that tasks are evenly distributed to each node. This load balancing method prevents inconsistent data processing due to overload of a certain node and ensures the reliability of processing results.

[0139] Paxos protocol: The Paxos protocol is used for consistency maintenance to ensure that data updates between multiple nodes are negotiated and agreed upon. Only data updates agreed upon by the majority of nodes will take effect, thus avoiding inconsistencies caused by data asynchrony between nodes.

[0140] Health status monitoring: Real-time monitoring of the health status of each node, timely detection and response to failures. This monitoring can ensure that when a node fails, the system can take prompt measures to avoid inconsistent data processing caused by node failure.

[0141] Data compression and optimization: Data compression can reduce the transmission volume and improve bandwidth utilization. At the same time, the consistent hashing algorithm is used to optimize the allocation of data and tasks. This measure effectively reduces the risk of data loss or damage during transmission and further maintains data consistency.

[0142] Optimize allocation and scheduling: Use consistent hashing algorithm to optimize the allocation of data and tasks, reduce data skew, and ensure uniform load on each node. This optimization strategy avoids processing anomalies caused by uneven data distribution and ensures that each node can accurately process the tasks assigned to it.

[0143] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and alterations may be made to the embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A neural network-based image data anomaly analysis system, characterized in that: The analysis system consists of a data preprocessing module, a data storage and management module, a data processing module, a data consistency and coordination module, a fault tolerance and recovery module, a performance optimization module, and a data transmission and scheduling module; The data preprocessing module is used to perform standardization processing on the image data and unify the data format and range; The data storage and management module is used to store the image data in pieces in a distributed file system; The data processing module uses a distributed computing framework to process image data in parallel, dynamically allocates tasks according to node load, and optimizes resource usage; The data consistency and coordination module uses the Paxos protocol to reach consensus through negotiation between multiple nodes to ensure that data updates of all nodes are consistent; The fault tolerance and recovery module is used to monitor node status in real time, detect and respond to faults; The performance optimization module uses data compression technology to reduce data transmission volume, improve network bandwidth utilization, and uses consistent hashing algorithm to optimize data and task allocation, thereby improving system scalability; The data transmission and scheduling module uses a consistent hashing algorithm to optimize the allocation of data and tasks, reduce data skew, and improve the scalability of the system; The parallel processing expression in the distributed computing in the data storage and management module is: Suppose there are N computing nodes and T image data tasks, where each task ti is assigned to node j, and the processing time of node j is P j (t i ), the total processing time of a task is the sum of the processing time of all tasks on node j; Among them: tasks j represents the set of all tasks assigned to node j; The total processing time of the global task is expressed as: Among them: nodes represents the set of all computing nodes.

2. The image data anomaly analysis system based on a neural network according to claim 1, characterized in that: The data storage and management module also includes node load dynamic allocation, and the node load dynamic allocation expression is: In dynamic load balancing, the distribution of tasks will be adjusted according to the current node load. Let the current load of each node j be L j , assign tasks T according to the load i To node j, load L j Defined according to the processing time and task volume of node j: Among them: New Task Processing Time is the processing time of the new task; To optimize resource usage, tasks are distributed by minimizing node load: Where: t i is the task to be assigned; node j is the target node; L j is the load of node j.

3. The image data anomaly analysis system based on a neural network according to claim 1, characterized in that: The purpose of the standardization process in the data preprocessing module is to adjust the data to a distribution with a mean of 0 and a standard deviation of 1. Assume that each pixel value x of the image data needs to be standardized, with a mean of μ and a standard deviation of σ: Where: x is the pixel value of the original image data; μ is the mean of the image data; σ is the standard deviation of the image data; x ′ is the normalized pixel value; The data preprocessing module also includes minimum-maximum standardization, expressed as: Minimum-maximum standardization scales the data to the specified range, [0,1], and sets the minimum value of the original data to x min , the maximum value is x max : Where: x is the pixel value of the original image data; x min is the minimum value of the image data; x max is the maximum value of the image data; x ′ is the normalized pixel value.

4. The image data anomaly analysis system based on a neural network according to claim 3, characterized in that: The data preprocessing module unifies the data format by converting the data into a specific format: converting the image data from RGB to a grayscale image, assuming that the pixel value of the RGB image is (R, G, B), and converting to a grayscale value G using the following formula: G=0.2989*R+0.5870*G+0.1140*B Where: R is the pixel value of the red channel; G is the pixel value of the green channel; B is the pixel value of the blue channel; G is the pixel value of the grayscale image; Before the standardization process, the mean and standard deviation need to be calculated. Suppose the image data has a total of N pixel values ​​x i : Where: μ is the mean value of the image data; is the sum of all pixel values; N is the total number of pixel values.

5. The image data anomaly analysis system based on neural network according to claim 4, characterized in that: The node load calculation expression in the data processing module is: L j =P j / T j Where: P j is the total processing time of the tasks processed on node j; T j is the number of tasks currently being processed on node j; L j is the load of node j.

6. The image data anomaly analysis system based on neural network according to claim 5, characterized in that: The load balancing target expression in the data processing module is: Where: L target is the ideal load level; Σ j∈nodes P j is the sum of processing time of all nodes; N is the number of nodes.

7. The image data anomaly analysis system based on neural network according to claim 6, characterized in that: The data processing module dynamically allocates tasks to achieve load balancing. Task allocation is optimized by minimizing the difference between each node load and the target load. The adjusted node load calculation formula is: Reassign Task t i to node j such that|L j -L target |is minimized Where: t i is the task to be reallocated; node j is the target node for receiving task t i ; L j is the load of node j; L target is the ideal load level; |L j -L target | is the absolute difference between the load of node j and the target load.

8. The image data anomaly analysis system based on neural network according to claim 7, characterized in that: The proposal number expression of the proposal in the data consistency and coordination module is: pi>pj(for all proposals i with proposal number pj accepted The expression of the preparation stage is: p is the current proposal number; p accept is the maximum proposal number that node j has accepted; v accept is the proposal value corresponding to node j; A j is the set of accepted proposals of node j; accept Stage: p>p accept Where: p is the current proposal number; p accept is the maximum proposal number accepted by node j.

9. The image data anomaly analysis system based on neural network according to claim 8, characterized in that: The node health status monitoring expression in the fault tolerance and recovery module is: S i =H i (t) Where: S i is the health status of node i at time t; H i (t): Health status function of node i at time t.

10. The image data anomaly analysis system based on neural network according to claim 9, characterized in that: The data compression expression in the performance optimization module is: CR=D original / D compressed Where: CR is the compression ratio; D original is the original data size D compressed The compressed data size.

Images