A data scanning method, device, apparatus and storage medium

Abstract

The application discloses a data scanning method and device, equipment and storage medium, and relates to the field of data processing. The method comprises the following steps: acquiring business transaction data, processing the business transaction data to obtain target data; performing segmentation processing on the target data by using a Bayesian optimization algorithm to obtain an optimal sharding threshold; generating a corresponding number of target sub-tables based on the numerical size of the optimal sharding threshold, and acquiring a plurality of data indexes corresponding to the target sub-tables; scanning data in the target sub-tables by using the data indexes and a preset scanning scheme, and performing clustering processing on the scanned data by using a self-organizing mapping neural network algorithm to obtain a final scanning result. In this way, the acquired data can be segmented by using the Bayesian optimization algorithm, and the scanned data can be clustered by using the self-organizing mapping neural network algorithm, so that the problem of data balance is solved, and the scanning computing power is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, in particular to a data scanning method and device, equipment and storage medium. BACKGROUND

[0002] Based on the demand of data-driven business whole process, it is necessary to scan massive transaction data in multiple dimensions and identify risk matters. With the gradual increase of the base of production transaction data, the existing technology cannot support the scanning of massive data.

[0003] The existing technical solution first performs data preprocessing according to the business scope in advance, then the data scanning service obtains all preprocessed data of the target data partition through single query, and then the data scanning application processes the suspicious points in memory according to the business indicators. After the suspicious point processing is completed, the suspicious data is compressed and uploaded to the electronic database for temporary storage, and the voucher address is downloaded, parsed and suspiciously stored through the message notification application program.

[0004] However, the existing technology cannot support massive data acquisition. Through single query, all preprocessed data of the target data partition is obtained. The larger the data volume of the target data partition is, the longer the application connection occupies time, resulting in data application connection timeout, which cannot support data scanning, often resulting in business demand compromise, reducing the data range of business indicator analysis and reducing the data scanning base. In addition, the single node scanning power efficiency in the existing technology is low, the application node fault tolerance is poor, and the computing capacity of the application cluster cannot be fully utilized, resulting in that the overall computing capacity cannot be effectively utilized, and the suspicious point processing time period is long. SUMMARY

[0005] Therefore, the purpose of the present application is to provide a data scanning method, device, equipment and storage medium, which can segment massive data based on the Bayesian optimization algorithm, obtain the optimal sharding threshold, solve the problem of uneven distribution of target table massive data, make the data scanning not limited by the base of the target data partition data, and can perform aggregated analysis on the target sharding data based on the self-organizing mapping neural network algorithm, improve the index scanning efficiency, shorten the data scanning period, and solve the problem of low efficiency of data scanning service computing power. The specific scheme is as follows:

[0006] In a first aspect, the present application discloses a data scanning method applied to a server, comprising:

[0007] Obtaining business transaction data, processing the business transaction data to obtain target data;

[0008] Segmenting the target data by a Bayesian optimization algorithm to obtain an optimal sharding threshold;

[0009] generate a corresponding number of target sub-tables based on the numerical size of the optimal sharding threshold, and obtain a plurality of data indexes corresponding to the plurality of target sub-tables;

[0010] scan data in the plurality of target sub-tables through the plurality of data indexes and a preset scanning scheme, and perform clustering processing on the scanned data through a self-organizing mapping neural network algorithm to obtain a final scanning result.

[0011] Optionally, the business transaction data is obtained, and the business transaction data is processed to obtain target data, including:

[0012] obtaining business transaction data in a target business system database, and performing data cleaning, conversion, screening, and logical deduplication on the business transaction data to obtain the target data.

[0013] Optionally, the target data is segmented through a Bayesian optimization algorithm to obtain an optimal sharding threshold, including:

[0014] determining a function value of a target fitness function and a Bayesian network model;

[0015] performing sampling processing on the target data through the function value of the target fitness function and the Bayesian network model to obtain the optimal sharding threshold.

[0016] Optionally, the function value of the target fitness function includes:

[0017] segmenting the target data according to a preset sharding range by using a hash modulo algorithm to obtain a plurality of sharding data;

[0018] performing variance value operation on each data in each of the sharding data to obtain a plurality of variances corresponding to the plurality of sharding data, and taking the plurality of variances as the function value of the target fitness function.

[0019] Optionally, after scanning data in the plurality of target sub-tables through the plurality of data indexes and a preset scanning scheme, the method further includes:

[0020] listening to suspicious information based on the data in the plurality of target sub-tables; the suspicious information includes suspicious detail analysis information, suspicious summary statistical information, and suspicious dimension statistical information;

[0021] generating a corresponding log monitoring file by using the suspicious information;

[0022] The log monitoring file is parsed, and if the parsing result indicates that the suspicious information is abnormal, the suspicious information is corrected, the log monitoring file is format-converted, and the converted log monitoring file is stored in a preset database;

[0023] If the parsing result indicates that the suspicious information is not abnormal, the log monitoring file is directly format-converted, and the converted log monitoring file is stored in the preset database.

[0024] Optionally, if the parsing result indicates that the suspicious information is abnormal, the correction of the suspicious information includes:

[0025] If the parsing result indicates that the suspicious information is abnormal, the corresponding correction data of the suspicious information is obtained through an abnormality monitoring and perception service, and the suspicious information is corrected through the correction data.

[0026] Optionally, the scanning of the data in the plurality of target sub-tables through the plurality of data indexes and a preset scanning scheme, and the clustering processing of the scanned data through a self-organizing mapping neural network algorithm to obtain the final scanning result include:

[0027] The scanning of the data in the plurality of target sub-tables is started through the data range, the data dimension, the data model, the scanning frequency in the preset scanning scheme, and the plurality of data indexes to obtain the scanned data.

[0028] The self-organizing mapping neural network algorithm is trained by using the scanned data, and the clustering processing of the scanned data is performed by using the trained self-organizing mapping neural network algorithm to obtain the final scanning result.

[0029] In a second aspect, the present application discloses a data scanning device applied to a server and comprising:

[0030] A data acquisition module is configured to acquire business transaction data, and process the business transaction data to obtain target data.

[0031] A threshold acquisition module is configured to perform segmentation processing on the target data through a Bayesian optimization algorithm to obtain an optimal sharding threshold.

[0032] An index acquisition module is configured to generate a corresponding number of target sub-tables based on the numerical size of the optimal sharding threshold, and acquire a plurality of data indexes corresponding to the target sub-tables.

[0033] The data processing module is configured to scan data in the target sub-tables through the data indexes and a preset scanning scheme, and to perform clustering processing on the scanned data through a self-organizing mapping neural network algorithm to obtain a final scanning result.

[0034] In a third aspect, the present application discloses an electronic device, comprising:

[0035] A memory configured to store a computer program;

[0036] A processor configured to execute the computer program to implement the data scanning method as described above.

[0037] In a fourth aspect, the present application discloses a computer readable storage medium configured to store a computer program, which, when executed by a processor, implements the data scanning method as described above.

[0038] In the present application, first, business transaction data is acquired, and the business transaction data is processed to obtain target data. Then, the target data is segmented through a Bayesian optimization algorithm to obtain an optimal segmentation threshold. Based on the numerical size of the optimal segmentation threshold, a corresponding number of target sub-tables are generated. The corresponding data indexes of the target sub-tables are acquired. Finally, data in the target sub-tables is scanned through the data indexes and a preset scanning scheme. The scanned data is clustered through a self-organizing mapping neural network algorithm to obtain a final scanning result. As can be seen, through the data scanning method in the present application, the business transaction data can be processed after being received. The processed data is segmented through a Bayesian optimization algorithm to obtain an optimal segmentation threshold. The target segmented data can be analyzed based on a self-organizing mapping neural network algorithm. In this way, the problem of uneven distribution of massive data in the target table can be solved. The data scanning is not limited by the cardinality of the target data partition data. The target segmented data can be analyzed based on a self-organizing mapping neural network algorithm. The scanning efficiency of the index is improved. The data scanning period is shortened. The problem of low efficiency of data scanning service computing power is solved. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application. Those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0040] Figure 1 A data scanning method flowchart is provided in the present application.

[0041] Figure 2 A specific data scanning method flowchart is provided for the present application;

[0042] Figure 3 A specific data scanning method flowchart is provided for the present application;

[0043] Figure 4 A specific data scanning method flowchart is provided for the present application;

[0044] Figure 5 A data scanning method overall structure diagram is provided for the present application;

[0045] Figure 6 A sensor internal structure diagram is provided for the present application;

[0046] Figure 7 A data scanning device structure schematic diagram is provided for the present application;

[0047] Figure 8 An electronic device structure diagram is provided for the present application. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0049] The existing technology can completely realize data scanning and data warehousing, but the existing technology cannot support mass data acquisition. The larger the data volume of the target data partition is, the longer the application connection occupancy time is, which cannot support data scanning. In addition, the single-node scanning computing power efficiency of the existing technology is low, the application node fault tolerance is poor, the computing capacity of the application cluster cannot be fully utilized, the overall computing power cannot be effectively utilized, and the processing time period is long.

[0050] In order to overcome the above technical problems, the present application provides a data scanning method, device, equipment and storage medium, which can segment mass data based on Bayesian optimization algorithm, obtain optimal sharding threshold, solve the problem of uneven distribution of target table mass data, make data scanning not limited by the cardinality of target data partition data, and can perform aggregation analysis on target sharding data based on self-organizing mapping neural network algorithm, improve index scanning efficiency, shorten data scanning period, and solve the problem of low efficiency of data scanning service computing power.

[0051] Referring to Figure 1As shown, the embodiment of the application discloses a data scanning method, applied to a server, comprising:

[0052] In step S11, business transaction data is acquired, and the business transaction data is processed to obtain target data.

[0053] In this embodiment, before the data is processed by using the Bayesian optimization algorithm and the self-organizing mapping neural network algorithm, the business transaction data needs to be acquired first, and then the business transaction data is processed to obtain target data that needs to be processed, and the specific process is as follows: the business transaction data in the target business system database is acquired, and the business transaction data is cleaned, converted, screened, and logically deduplicated to obtain the target data. That is, by configuring a big data extraction component connected to each business system database, the business transaction data is uniformly extracted to the mirror layer of the big data platform according to a period, and the business transaction data is cleaned, converted, screened, and logically deduplicated in the mirror layer to obtain the target data of the intermediate layer that needs to be processed by using the algorithm. It needs to be noted that cleaning the data refers to the process of correcting and deleting inaccurate data records from the database or data table, including identifying and replacing incomplete, inaccurate, irrelevant, or problematic data, and logically deduplicating the data is to clean the duplicate data, so as to avoid processing useless data by using the algorithm, and effectively improve the processing efficiency and accuracy of the data scanning method in the application.

[0054] In step S12, the target data is segmented by using the Bayesian optimization algorithm to obtain an optimal sharding threshold.

[0055] In this embodiment, the target data is segmented by using the Bayesian optimization algorithm to obtain an optimal sharding threshold. That is, after obtaining the target data of the intermediate layer that needs to be processed, the target data needs to be segmented in the intermediate layer by configuring an embedded function to refer to the Bayesian optimization algorithm to determine the optimal sharding threshold, and the segmentation of the target data by using the Bayesian optimization algorithm mainly includes determining the fitness function value, constructing the Bayesian model, and sampling the target data by using the function value of the target fitness function and the Bayesian network model to obtain the optimal sharding threshold. It needs to be noted that the data sharding refers to the process of decomposing a big data table containing data into a plurality of small tables, and the obtained sharding is distributed on a plurality of database cluster nodes, each sharding contains a subset of the original total data set, and a balancer is used to balance each sharding. In this way, the determination of the optimal sharding threshold can determine the number of data sharding.

[0056] Step S13, generating a corresponding number of target sub-tables based on the numerical size of the optimal sharding threshold, and obtaining a plurality of data indexes corresponding to the plurality of target sub-tables.

[0057] In this embodiment, a corresponding number of target sub-tables is generated based on the numerical size of the optimal sharding threshold, and a plurality of data indexes corresponding to the plurality of target sub-tables is obtained. That is, the target data is sharded by the numerical size of the obtained optimal sharding threshold, and a number of sub-tables corresponding to the numerical size of the optimal sharding threshold is automatically constructed in the model layer. In the process of constructing the data sub-table, the data corresponding to the intermediate layer and the target sub-table needs to be put into the target sub-table corresponding to the model layer, and the sharding record index information of the target sub-table is saved, so as to track the target sub-table based on the data index for subsequent data scanning service. In this way, when a large amount of data scanning occurs, the data can be sharded to solve the problem of uneven data distribution.

[0058] Step S14, scanning the data in the plurality of target sub-tables through the plurality of data indexes and a preset scanning scheme, and performing clustering processing on the scanned data through a self-organizing mapping neural network algorithm to obtain a final scanning result.

[0059] In this embodiment, after it is determined that the target sub-table has been generated based on the business transaction data, data scanning needs to be started according to the data range, data dimension, data model, and scanning frequency in the specified scanning scheme and the data index corresponding to the target sub-table. It needs to be noted that each scanning service in the scanning cluster is only for all data in the specified target sub-table shard, that is, each scanning service is for data in a specific target sub-table, and then a scanning engine is started to perform clustering analysis on the data through a self-organizing mapping neural network algorithm. It further needs to be noted that the clustering analysis of the data through the self-organizing mapping neural network algorithm can be divided into two steps. The first step is to randomly select scanned data and train the self-organizing mapping neural network using the selected scanned data. The second step is to cluster similar data. In this way, clustering the target sharded data through the self-organizing mapping neural network algorithm can improve the index scanning efficiency and shorten the data scanning period.

[0060] It can be seen that, first, the business transaction data is acquired, the business transaction data is processed to obtain target data, then the target data is segmented by a Bayesian optimization algorithm to obtain an optimal sharding threshold, a corresponding number of target sub-tables are generated based on the numerical size of the optimal sharding threshold, and a plurality of data indexes corresponding to the plurality of target sub-tables are acquired, finally, the data in the plurality of target sub-tables is scanned through the plurality of data indexes and a preset scanning scheme, and the scanned data is clustered by a self-organizing mapping neural network algorithm to obtain a final scanning result. It can be seen that, by the data scanning method, the business transaction data can be processed after the business transaction data is received, the processed data is segmented by the Bayesian optimization algorithm to obtain an optimal sharding threshold, and the target sharding data can be aggregated and analyzed based on the self-organizing mapping neural network algorithm. In this way, the problem of uneven distribution of massive data of the target table can be solved, the data scanning is not limited by the cardinality of the target data partition data, the target sharding data can be aggregated and analyzed based on the self-organizing mapping neural network algorithm, the scanning efficiency of the index is improved, the data scanning period is shortened, and the problem of low efficiency of data scanning service computing power is solved.

[0061] Based on the previous embodiment, the processed business transaction data needs to be segmented by the Bayesian optimization algorithm. Therefore, this embodiment describes in detail how to segment the data by the Bayesian optimization algorithm, as shown in Figure 2 The data scanning method disclosed by the embodiment of the application comprises the following steps:

[0062] In step S21, the business transaction data is acquired, and the business transaction data is processed to obtain target data.

[0063] In step S22, the function value of the target fitness function and the Bayesian network model are determined.

[0064] In the embodiment, the determination process of the target fitness function value is as follows: the target data is divided into several piece data according to a preset piece range by using a hash modulo algorithm; the variance of each data in each piece data is calculated to obtain several variances corresponding to the several piece data, and the several variances are taken as the function value of the target fitness function. That is, the target data is first hashed and moduloed, and the target data is divided into pieces according to the preset piece range 8 to 255 in the modulo process. It should be noted that the number of nodes of the Bayesian network, that is, the number of pieces of the target data, is set to 8 by default. After determining the number of pieces of the target data, the fitness function and the function value of the fitness function need to be determined. In the embodiment, the fitness function is a variance value formula of the number of data in each piece data, and the variance value formula is as follows:

[0065]

[0066] where S 2 is the total variance, p i is the number of individual pieces, u is the total mean, and n is the number of pieces.

[0067] After determining the fitness function, the initial population r(t) is randomly selected from the range of the number of pieces in the m piece range, and t=0, where t is the iteration number. The m pieces are taken as the partition threshold to divide the target data m times.

[0068] In the embodiment, the construction process of the Bayesian network model is as follows: the individual with a larger function value is selected from the population r(t) as a sample set, and the Bayesian network parameters of the individual are calculated. Then, the Bayesian network is sampled and processed through the conditional probability distribution of each node to generate a new population s(t). It should be noted that the population s(t) generated by the individual selected according to different function values is different. All s(t) populations are collected, and the new population x(t) is formed by using all s(t) populations. The function value of each individual in the population is calculated, the individual with the largest function value is found, and the individual with the largest function value is taken as a vaccine. The population x(t) is vaccinated through a preset vaccination probability to update the population x(t). The m individuals with the largest function value are selected from the updated population to form a new population r(t+1), and t=t+1 at the same time to realize the update operation of the next population x(t).

[0069] In step S23, the target data is sampled and processed by the function value of the target fitness function and the Bayesian network model to obtain the optimal piece threshold.

[0070] In this embodiment, the target data is sampled by the function value of the target fitness function and the Bayesian network model to obtain the optimal fragmentation threshold. That is, the target data is processed by using the constructed Bayesian network model and the function value of the fitness function, and it is determined whether the fragmentation threshold obtained after processing meets the preset termination condition. If yes, the data scanning operation is performed; if not, the optimal fragmentation threshold is determined again.

[0071] In step S24, a number of target sub-tables corresponding to the number of the optimal fragmentation threshold are generated, and a number of data indexes corresponding to the number of target sub-tables are obtained.

[0072] In step S25, the data in the number of target sub-tables is scanned by using the number of data indexes and a preset scanning scheme, and the scanned data is clustered by using a self-organizing mapping neural network algorithm to obtain a final scanning result.

[0073] It should be noted that the more detailed processing method of step S21, step S24 and step S25 can refer to the foregoing embodiments, which will not be described here.

[0074] As can be seen, in this embodiment, the business transaction data is first obtained, the business transaction data is processed to obtain target data, the function value of the target fitness function and the Bayesian network model are determined, the target data is sampled by the function value of the target fitness function and the Bayesian network model to obtain the optimal fragmentation threshold, and finally a number of target sub-tables corresponding to the number of the optimal fragmentation threshold are generated, a number of data indexes corresponding to the number of target sub-tables are obtained, the data in the number of target sub-tables is scanned by using the number of data indexes and a preset scanning scheme, and the scanned data is clustered by using a self-organizing mapping neural network algorithm to obtain a final scanning result. In this way, the massive data can be segmented based on the Bayesian optimization algorithm to obtain the optimal fragmentation threshold, the problem of uneven distribution of target table massive data is solved, and the data scanning is not limited by the cardinality of the target data partition data.

[0075] Based on the foregoing embodiments, after the optimal fragmentation threshold is determined, the data in the sub-table needs to be scanned, and the scanned data needs to be clustered by using a self-organizing mapping neural network algorithm. Therefore, this embodiment makes a detailed description of how to cluster the scanned data by using a self-organizing mapping neural network algorithm, which is described in detail in the foregoing embodiment. Figure 3 As shown in the foregoing embodiment, the embodiment of the application discloses a data scanning method, which comprises:

[0076] Step S31, obtaining business transaction data, processing the business transaction data to obtain target data.

[0077] Step S32, performing segmentation processing on the target data by a Bayesian optimization algorithm to obtain an optimal segmentation threshold.

[0078] Step S33, generating a corresponding number of target sub-tables based on the numerical size of the optimal segmentation threshold, and obtaining a plurality of data indexes corresponding to the plurality of target sub-tables.

[0079] Step S34, starting scanning of data in the plurality of target sub-tables by the data range, data dimension, data model, scanning frequency in the preset scanning scheme, and the plurality of data indexes to obtain scanned data.

[0080] In this embodiment, before starting data scanning, the data indexes generated in the process of generating data sub-tables in the model layer need to be obtained, and the model semantic information of the data model in the preset scanning scheme needs to be obtained and loaded in the memory for preheating. After preheating is completed, the data range, data dimension, data model, scanning frequency in the preset scanning scheme, and the plurality of data indexes corresponding to the data sub-tables in the model layer are used as parameters of the scanning engine to start the scanning engine and start the distributed scanning service. It should be noted that each scanning service in the scanning cluster only loads all data in the specified data sub-table, and starts the index scanning engine in combination with the index set information of the model semantic, and performs aggregation analysis by referring to the self-organizing mapping neural network algorithm, and takes the target data as the input layer and the index condition as the hidden layer.

[0081] Step S35, training the self-organizing mapping neural network algorithm using the scanned data, and performing clustering processing on the scanned data using the trained self-organizing mapping neural network algorithm to obtain the final scanning result.

[0082] In this embodiment, the clustering processing of the scanned data by the self-organizing mapping neural network algorithm mainly includes two stages. The first stage is the learning stage of the self-organizing mapping neural network algorithm, which needs to randomly select data from the scanned data as training data to train the self-organizing mapping neural network, select a winning neuron according to the Euclidean distance, and update the weights of the winning neuron and its neighborhood neurons. The second stage is the clustering stage, which needs to map the scanned data as test data to neurons, and similar data will be mapped to adjacent neurons to achieve clustering processing of the data.

[0083] It needs to be pointed out that the specific implementation of the above two stages is as follows: first, the network initialization of the self-organizing mapping neural network needs to be carried out, the initial value of the weight vector w ij between the input layer and the mapping layer is set by using random numbers, then the input vector x i is input to the input layer, where i represents the number of data sub-tables generated based on target data, and the distance between the weight vector of the mapping layer and the input vector, the distance between the neuron of the mapping layer and the input vector are calculated, and the formula is as follows:

[0084]

[0085] Where d j is the distance between the neuron of the mapping layer and the input vector, x i is the input vector, and w ij is the weight vector.

[0086] The neuron whose distance to the input vector is the smallest is calculated and selected, the neuron whose distance to the input vector is the smallest is called the winning neuron, and its adjacent neuron set is given, then the weights of the winning neuron and the neurons located in its adjacent neurons are updated according to the following formula:

[0087] w ij (t+1)=w ij (t)+w ij

[0088] Where w ij (t+1) represents the updated winning neuron weight, w ij (t) represents the current winning neuron weight, and w ij represents the preset learning force.

[0089] Finally, it is judged whether the updated winning neuron weight meets the preset computing power requirement, if not, the winning neuron weight is updated again, if the preset computing power requirement is met, the algorithm is ended, and the scanned data is mapped to the neuron as test data to realize the clustering processing of the data.

[0090] It needs to be pointed out that the more detailed processing method of step S31, step S32 and step S33 can refer to the foregoing embodiments, which will not be repeated here.

[0091] It can be seen that, in the embodiment, first, the business transaction data is acquired, the business transaction data is processed to obtain target data, the target data is segmented by a Bayesian optimization algorithm to obtain an optimal segmentation threshold, a corresponding number of target sub-tables are generated based on the numerical size of the optimal segmentation threshold, and a plurality of data indexes corresponding to the plurality of target sub-tables are acquired, finally, the data in the plurality of target sub-tables is scanned by using the data range, the data dimension, the data model, the scanning frequency in the preset scanning scheme and the plurality of data indexes to obtain scanned data, the self-organizing mapping neural network algorithm is trained by using the scanned data, and the scanned data is clustered by using the trained self-organizing mapping neural network algorithm to obtain a final scanning result. In this way, the target segmentation data can be aggregated and analyzed based on the self-organizing mapping neural network algorithm, the index scanning efficiency is improved, the data scanning period is shortened, and the problem of low data scanning service computing power is solved.

[0092] In the present application, in addition to the segmentation of the target data by the Bayesian optimization algorithm and the clustering of the scanned data by the self-organizing mapping neural network algorithm, the process and the suspicious information can also be monitored, and the corresponding log monitoring file can be generated by using the sensor, and the data can be corrected. Therefore, the present embodiment describes in detail how to monitor the suspicious information and generate the corresponding log monitoring file by using the sensor, and correct the data. See Figure 4 The data scanning method disclosed by the embodiment comprises the following steps:

[0093] In step S41, suspicious information based on the data in the plurality of target sub-tables is monitored; the suspicious information comprises suspicious detail analysis information, suspicious summary statistical information, and suspicious dimension statistical information.

[0094] In the present embodiment, the suspicious information based on the data in the plurality of target sub-tables is monitored. That is, as shown in Figure 5 When the scanning service aggregates data by using the self-organizing mapping neural network, suspicious information is generated, and the suspicious information comprises suspicious detail analysis information, suspicious summary statistical information, and suspicious dimension statistical information. The overall computing capacity of the suspicious information cluster can be used to monitor the suspicious detail analysis message, the suspicious summary statistical message, and the suspicious dimension statistical message respectively.

[0095] In step S42, the corresponding log monitoring file is generated by using the suspicious information.

[0096] In the present embodiment, the corresponding log monitoring file is generated by using the suspicious information. That is, in the main process of data scanning and suspicious information monitoring, as shown in Figure 6The shown perceiver can capture abnormal monitoring logs and execution process logs of service running through the configuration service provider aspect, and generate corresponding log monitoring files, so as to process the suspicious information through analysis of the log monitoring files.

[0097] Step S43, the log monitoring file is analyzed, if the analysis result indicates that the suspicious information is abnormal, the suspicious information is corrected, the log monitoring file is format-converted, and the converted log monitoring file is stored in the preset database.

[0098] In the embodiment, the log monitoring file is analyzed, if the analysis result indicates that the suspicious information is abnormal, the suspicious information is corrected, the log monitoring file is format-converted, and the converted log monitoring file is stored in the preset database. That is, the perceiver listens to the incremental abnormal monitoring log file and the execution process log file through the log extraction component deployed on each application node, analyzes the log file, and after data format conversion, puts it into the database of the perceiver for saving, and if the analysis result indicates that the suspicious information is abnormal, the suspicious information is corrected. The specific implementation is as follows: if the analysis result indicates that the suspicious information is abnormal, the correction data corresponding to the suspicious information is obtained through the abnormal monitoring perception service, and the suspicious information is corrected through the correction data. That is, the timer in the perceiver triggers once every 5 minutes according to the period, and calls the abnormal monitoring perception service to obtain correction data, and it needs to be noted that the correction data can also be obtained manually, if there is correction data, the automatic correction message is sent, the risk identification application subscribes to the correction message and consumes it, and the data correction of the corresponding link is performed according to the execution process.

[0099] Step S44, if the analysis result indicates that the suspicious information is not abnormal, the log monitoring file is directly format-converted, and the converted log monitoring file is stored in the preset database.

[0100] In the embodiment, if the analysis result indicates that the suspicious information is not abnormal, the log monitoring file is directly format-converted, and the converted log monitoring file is stored in the preset database. That is, if the analysis result of the log file indicates that the suspicious information is normal, the correction data for abnormal suspicious information does not need to be obtained, the log monitoring file can be directly format-converted, and the converted log monitoring file is stored in the preset database. In this way, the abnormal situation in the data scanning process can be captured and corrected in time, and the reliability of the data scanning method in the application is effectively improved.

[0101] It can be seen that in the embodiment, firstly, suspicious information based on data in a plurality of target sub-tables is listened to, then the suspicious information is used to generate a corresponding log monitoring file, and the log monitoring file is parsed, if the parsing result represents that the suspicious information is abnormal, the suspicious information is corrected, and the log monitoring file is format-converted, and the converted log monitoring file is stored in a preset database, finally, if the parsing result represents that the suspicious information is not abnormal, the log monitoring file is directly format-converted, and the converted log monitoring file is stored in the preset database. In this way, the abnormality in the data scanning task, the suspicious analysis message and the like can be captured, compensated and corrected in time according to the abnormality monitoring perceiver of the log middleware.

[0102] Referring to Figure 7 The embodiment of the application discloses a data scanning device applied to a server end, comprising:

[0103] A data acquisition module 11 is configured to acquire business transaction data, and process the business transaction data to obtain target data.

[0104] A threshold acquisition module 12 is configured to perform segmentation processing on the target data by using a Bayesian optimization algorithm to obtain an optimal sharding threshold.

[0105] An index acquisition module 13 is configured to generate a corresponding number of target sub-tables based on the numerical size of the optimal sharding threshold, and acquire a plurality of data indexes corresponding to the target sub-tables.

[0106] A data processing module 14 is configured to scan data in the target sub-tables by using the plurality of data indexes and a preset scanning scheme, and perform clustering processing on the scanned data by using a self-organizing mapping neural network algorithm to obtain a final scanning result.

[0107] It can be seen that, first, the business transaction data is acquired, the business transaction data is processed to obtain target data, then the target data is segmented by a Bayesian optimization algorithm to obtain an optimal sharding threshold, and a corresponding number of target sub-tables are generated based on the numerical size of the optimal sharding threshold, and a plurality of data indexes corresponding to the plurality of target sub-tables are acquired, finally, the data in the plurality of target sub-tables is scanned through the plurality of data indexes and a preset scanning scheme, and the scanned data is clustered by a self-organizing mapping neural network algorithm to obtain a final scanning result. It can be seen that, by the data scanning method, the business transaction data can be processed after receiving the business transaction data, the processed data can be segmented by a Bayesian optimization algorithm to obtain an optimal sharding threshold, and the target sharding data can be aggregated and analyzed based on a self-organizing mapping neural network algorithm. In this way, the problem of uneven distribution of massive data of the target table can be solved, the data scanning is not limited by the cardinality of the target data partition data, the target sharding data can be aggregated and analyzed based on a self-organizing mapping neural network algorithm, the scanning efficiency of the index is improved, the data scanning period is shortened, and the problem of low efficiency of data scanning service computing power is solved.

[0108] In some embodiments, the data acquisition module 11 can specifically include:

[0109] The data processing unit is configured to acquire business transaction data in a target business system database, and perform data cleaning, conversion, filtering, and logical deduplication on the business transaction data to obtain the target data.

[0110] In some embodiments, the threshold acquisition module 12 can specifically include:

[0111] The model construction submodule is configured to determine a function value of a target fitness function and a Bayesian network model.

[0112] The threshold determination submodule is configured to sample the target data by the function value of the target fitness function and the Bayesian network model to obtain the optimal sharding threshold.

[0113] In some embodiments, the model construction submodule can specifically include:

[0114] The data sharding unit is configured to shard the target data according to a preset sharding range by a hash modulo algorithm to obtain a plurality of sharded data.

[0115] The function value determination unit is configured to perform variance value operation on each data in each of the sharded data to obtain a plurality of variances corresponding to the plurality of sharded data, and take the plurality of variances as the function value of the target fitness function.

[0116] In some embodiments, the data scanning device can further include:

[0117] an information monitoring module configured to monitor suspicious information based on the data in the target sub-tables; the suspicious information includes suspicious detailed analysis information, suspicious summary statistical information, and suspicious dimension statistical information;

[0118] a monitoring file generation module configured to generate a corresponding log monitoring file using the suspicious information;

[0119] a data conversion module configured to parse the log monitoring file, correct the suspicious information if the parsing result indicates that the suspicious information is abnormal, format-convert the log monitoring file, and store the converted log monitoring file in a preset database;

[0120] a data storage module configured to format-convert the log monitoring file directly if the parsing result indicates that the suspicious information is not abnormal, and store the converted log monitoring file in the preset database.

[0121] In some embodiments, the data conversion module can specifically include:

[0122] a data correction unit configured to acquire correction data corresponding to the suspicious information through an abnormality monitoring and perception service if the parsing result indicates that the suspicious information is abnormal, and correct the suspicious information through the correction data.

[0123] In some embodiments, the data processing module 14 can specifically include:

[0124] a data scanning unit configured to scan the data in the target sub-tables through data ranges, data dimensions, data models, scanning frequencies, and the plurality of data indexes in the preset scanning scheme to obtain scanned data;

[0125] a data clustering unit configured to train the self-organizing mapping neural network algorithm using the scanned data, and perform clustering processing on the scanned data using the trained self-organizing mapping neural network algorithm to obtain a final scanning result.

[0126] Further, the embodiments of the present application also disclose an electronic device, Figure 8 is a structural diagram of an electronic device 20 according to an exemplary embodiment, and the content in the figure cannot be considered as any limitation on the use range of the present application.

[0127] Figure 8A structural schematic diagram of an electronic device 20 is provided in the embodiments of the present application. The electronic device 20 can specifically include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26. The memory 22 is configured to store a computer program, and the processor 21 is configured to load and execute the computer program to implement the related steps in the data scanning method disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in the embodiments of the present application can be specifically an electronic computer.

[0128] In the embodiments of the present application, the power supply 23 is configured to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 is capable of creating a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which is not limited specifically herein; the input / output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which is not limited specifically herein.

[0129] In addition, the memory 22 as a carrier for resource storage can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage mode can be temporary storage or permanent storage.

[0130] The operating system 221 is configured to manage and control each hardware device on the electronic device 20 and the computer program 222, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the data scanning method executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 can further include a computer program capable of completing other specific work.

[0131] Further, the present application further discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the foregoing disclosed data scanning method. The specific steps of the method can refer to the corresponding contents disclosed in the foregoing embodiments, which will not be repeated here.

[0132] The embodiments in the present specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can refer to the method part.

[0133] Those skilled in the art will further appreciate that the units and algorithm steps of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various examples have been described herein in terms of their functionality, which has been described generally and symbolically in flow charts. Having thus described the functionality of the examples, a person of ordinary skill in the art will be able to implement such functionality in hardware and / or software, and will recognize that the bounds of the examples are not limited by one approach or the other. The various examples can be realized in a centralized fashion in one computer system or network, or in a distributed fashion where different elements are spread across several computer systems or sub-networks. Any kind of computer system or other apparatus adapted for carrying out the methods described herein is suited. A typical combination of hardware and software could be a general purpose computer system with a computer program that, when being loaded and executed, carries out the methods described herein.

[0134] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in random access memory (RAM), flash memory, read-only memory (ROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, hard disk can be used as a storage medium.

[0135] Finally, it should be noted that the terms "first", "second", and the like, herein do not denote any order, quantity, combination, or importance, but rather are used to distinguish one element from another, and are more especially used for the purpose of illustration and not of limitation. Also, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. The terms "a" or "an", as used herein, mean "one or more".

[0136] The above detailed description has set forth various examples of the technology disclosed, and explained the principles and operation of the examples as set forth in the application. The examples are only examples and are not intended to limit the scope of the application. One skilled in the art will be able to implement the described technology with the disclosed material.

Claims

1. A data scanning method, characterized by, Applied to a server, comprising: Obtaining business transaction data, processing the business transaction data to obtain target data; Segmenting the target data by a Bayesian optimization algorithm to obtain an optimal sharding threshold; Based on the numerical size of the optimal sharding threshold, a corresponding number of target sub-tables are generated, and a plurality of data indexes corresponding to the plurality of target sub-tables are obtained; Scan the data in the plurality of target sub-tables through the plurality of data indexes and a preset scanning scheme, and cluster the scanned data through a self-organizing mapping neural network algorithm to obtain a final scanning result; Wherein, the target data is segmented by the Bayesian optimization algorithm to obtain the optimal sharding threshold, comprising: Determine the function value of the target fitness function and the Bayesian network model; Sample the target data by the function value of the target fitness function and the Bayesian network model to obtain the optimal sharding threshold; Wherein, the function value of the target fitness function comprises: Sharding the target data according to the preset sharding range by using the hash modulo algorithm to obtain a plurality of sharding data; Respectively, each data in each sharding data is subjected to variance value operation to obtain a plurality of variances corresponding to the plurality of sharding data, and the plurality of variances are taken as the function value of the target fitness function; Wherein, the scanning of the data in the plurality of target sub-tables through the plurality of data indexes and the preset scanning scheme, and the clustering of the scanned data through the self-organizing mapping neural network algorithm to obtain the final scanning result, comprising: Scan the data in the plurality of target sub-tables through the data range, data dimension, data model, scanning frequency in the preset scanning scheme and the plurality of data indexes to obtain scanned data; Train the self-organizing mapping neural network algorithm using the scanned data, and cluster the scanned data using the trained self-organizing mapping neural network algorithm to obtain the final scanning result.

2. The data scanning method of claim 1, wherein, The target data is obtained by processing the business transaction data, comprising: Obtaining business transaction data in a target business system database, and performing data cleaning, conversion, filtering, and logical deduplication on the business transaction data to obtain the target data.

3. The data scanning method of claim 1, wherein, After scanning the data in the plurality of target sub-tables through the plurality of data indexes and the preset scanning scheme, further comprising: Listen to suspicious information based on the data in the plurality of target sub-tables; the suspicious information includes suspicious detail analysis information, suspicious summary statistical information, and suspicious dimension statistical information; Generate a corresponding log monitoring file using the suspicious information; Parse the log monitoring file, if the parsing result indicates that the suspicious information is abnormal, correct the suspicious information, and format convert the log monitoring file, and store the converted log monitoring file in a preset database; If the analysis result represents that the suspicious information is normal, the log monitoring file is directly converted in format, and the converted log monitoring file is stored in the preset database.

4. The data scanning method of claim 3, wherein, If the analysis result represents that the suspicious information is abnormal, the suspicious information is corrected, including: If the analysis result represents that the suspicious information is abnormal, the correction data corresponding to the suspicious information is obtained through the abnormal monitoring and perception service, and the suspicious information is corrected through the correction data.

5. A data scanning device, characterized by Applied to a service end, comprising: A data acquisition module is configured to acquire business transaction data, and process the business transaction data to obtain target data; A threshold acquisition module is configured to perform segmentation processing on the target data through a Bayesian optimization algorithm to obtain an optimal sharding threshold; An index acquisition module is configured to generate a corresponding number of target sub-tables based on the numerical size of the optimal sharding threshold, and acquire a plurality of data indexes corresponding to the target sub-tables; A data processing module is configured to scan data in the target sub-tables through the plurality of data indexes and a preset scanning scheme, and perform clustering processing on the scanned data through a self-organizing mapping neural network algorithm to obtain a final scanning result. The threshold acquisition module comprises: A model construction submodule is configured to determine a function value of a target fitness function and a Bayesian network model; A threshold determination submodule is configured to perform sampling processing on the target data through the function value of the target fitness function and the Bayesian network model to obtain the optimal sharding threshold. The model construction submodule comprises: A data sharding unit is configured to perform sharding on the target data according to a preset sharding range through a hash modulo algorithm to obtain a plurality of sharded data; A function value determination unit is configured to perform variance value operation on each data in each of the sharded data to obtain a plurality of variances corresponding to the plurality of sharded data, and take the plurality of variances as the function value of the target fitness function. The data processing module comprises: A data scanning unit is configured to perform scanning on the data in the target sub-tables through a data range, a data dimension, a data model, a scanning frequency in the preset scanning scheme, and the plurality of data indexes to obtain scanned data; A data clustering unit is configured to train the self-organizing mapping neural network algorithm using the scanned data, and perform clustering processing on the scanned data using the trained self-organizing mapping neural network algorithm to obtain a final scanning result.

6. An electronic device, comprising: Comprise: A memory is configured to save a computer program; A processor is configured to execute the computer program to implement the data scanning method according to any one of claims 1 to 4.

7. A computer readable storage medium characterized in that, A computer program is saved, and the computer program is executed by a processor to implement the data scanning method according to any one of claims 1 to 4.

Images