Transaction data processing method and device, equipment and storage medium
By performing abnormal detection, cleaning and clustering analysis on transaction data, the problem of low quality of transaction data is solved, and the high quality and security of data are improved.
Patent Information
- Application Number
- CN202411705924.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-26
- Publication Date
- 2025-01-10
AI Technical Summary
The existing technology cannot improve the data quality of transaction data, resulting in poor data quality.
By obtaining the original transaction data for data abnormality detection, if there is no abnormality, data cleaning and clustering analysis are performed, and finally encrypted based on the cluster analysis results to obtain the target transaction data.
The data quality of transaction data is significantly improved, and dirty and inconsistent data are removed through abnormal detection and secondary processing, improving the accuracy and security of the data.
Smart Images
Figure CN119293031A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a transaction data processing method, device, equipment and storage medium. Background Art
[0002] With the continuous development of modern technology and digital technology in the information age, data has gradually become an important and core resource in the current era. With the rapid increase in the scale of data, problems related to data processing have arisen, especially for transaction data. How to ensure the security and effectiveness of transaction data has become an urgent problem to be solved.
[0003] Currently, the main method for processing transaction data is to collect and analyze the data before storing it. However, this method cannot improve the data quality of transaction data and there is a problem of poor data quality.
[0004] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention
[0005] The main purpose of the present invention is to provide a transaction data processing method, device, equipment and storage medium, aiming to solve the technical problem that the current transaction data processing method cannot improve the data quality of transaction data and has poor data quality.
[0006] To achieve the above object, the present invention provides a transaction data processing method, which comprises the following steps:
[0007] Get raw transaction data;
[0008] Performing data anomaly detection on the original transaction data to obtain a detection result;
[0009] Determine whether there is abnormal transaction data in the original transaction data based on the detection result, and if there is no abnormal transaction data, process the original transaction data once to obtain reference transaction data;
[0010] Performing cluster analysis on the reference transaction data;
[0011] The reference transaction data is processed secondary based on the cluster analysis result to obtain target transaction data.
[0012] In some embodiments, performing data anomaly detection on the original transaction data to obtain a detection result includes:
[0013] Acquire a historical transaction data set, and input features of the historical transaction data set into a preset neural network model;
[0014] Initializing the weights and biases of the learning parameters, and updating the cell states of the preset neural network model during the model training process according to the characteristics of the historical transaction data set and the learning parameters;
[0015] After completing the cell state update, using a decision function in the output layer of the preset neural network model to determine the prediction result of the preset neural network model, and constructing a corresponding loss function based on the prediction result;
[0016] Determine whether the loss function is within a preset range. If the loss function is not within the preset range, optimize the weight and bias of the learning parameter to adjust the learning parameter, and return to execute the step of updating the cell state of the preset neural network model during the model training process according to the characteristics of the historical transaction data set and the learning parameter, until the loss function is within the preset range, so as to complete the model training and obtain the abnormal data detection model;
[0017] The original transaction data is input into the abnormal data detection model to obtain a detection result.
[0018] In some embodiments, the expression of the decision function is f(x)=sign(wx+b), where the dimension of the weight vector w is the same as the dimension of the input feature x, and each element w in w i Represents the corresponding feature x i The weight in the classification decision, b is the bias term,
[0019] The expression of the loss function is Among them, y is the predicted classification label corresponding to the predicted result.
[0020] In some embodiments, the processing of the original transaction data to obtain reference transaction data includes:
[0021] Extracting metadata about data attributes of the original transaction data;
[0022] identifying dirty data in metadata of the data attributes;
[0023] Determine a data representation form of the dirty data;
[0024] Determining a data cleaning strategy based on the representation;
[0025] The original transaction data is cleaned based on the data cleaning strategy to obtain reference transaction data.
[0026] In some embodiments, the cleaning strategy includes a first cleaning strategy, a second cleaning strategy and a third cleaning strategy, wherein the first cleaning strategy is used to clean null values, the second cleaning strategy is used to clean erroneous values, and the third cleaning strategy is used to clean inconsistent data.
[0027] In some embodiments, performing cluster analysis on the reference transaction data includes:
[0028] Starting from a SOM layer of size 2×2 in the self-organizing map network, repeatedly inputting training sample vectors into the self-organizing map network, and training the SOM layer of the self-organizing map network until a preset number of iterations is reached;
[0029] Detect whether the current SOM layer meets the preset horizontal expansion criterion function;
[0030] If the preset horizontal expansion criterion function is satisfied, then continue to detect and check whether each neuron in the current SOM layer satisfies the preset vertical expansion criterion function;
[0031] If the preset vertical expansion criterion function is satisfied, it is determined that the self-organizing map network training is completed to obtain a clustering model;
[0032] The reference transaction data is input into the clustering model to obtain a clustering analysis result.
[0033] In some embodiments, the secondary processing of the reference transaction data based on the cluster analysis result to obtain the target transaction data includes:
[0034] Determining data classification of the reference transaction data based on the cluster analysis result;
[0035] determining a data encryption level according to the data classification;
[0036] Determining a corresponding encryption method based on the data encryption level;
[0037] The reference transaction data is encrypted according to the encryption method to obtain target transaction data.
[0038] In addition, to achieve the above-mentioned purpose, the present invention further proposes a transaction data processing device, the transaction data processing device comprising:
[0039] Acquisition module, used to obtain original transaction data;
[0040] A detection module, used to perform data anomaly detection on the original transaction data to obtain a detection result;
[0041] a processing module, configured to determine whether there is abnormal transaction data in the original transaction data based on the detection result, and if there is no abnormal transaction data, process the original transaction data once to obtain reference transaction data;
[0042] An analysis module, used for performing cluster analysis on the reference transaction data;
[0043] The processing module is used to perform secondary processing on the reference transaction data based on the cluster analysis result to obtain target transaction data.
[0044] In addition, to achieve the above-mentioned purpose, the present invention also proposes a transaction data processing device, which includes: a memory, a processor, and a transaction data processing program stored in the memory and executable on the processor, and the transaction data processing program is configured to implement the steps of the transaction data processing method described above.
[0045] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which a transaction data processing program is stored, and when the transaction data processing program is executed by a processor, the steps of the transaction data processing method described above are implemented.
[0046] The present invention obtains original transaction data; performs data anomaly detection on the original transaction data to obtain a detection result; determines whether there is abnormal transaction data in the original transaction data based on the detection result, and if there is no abnormal transaction data, performs a primary processing on the original transaction data to obtain reference transaction data; performs cluster analysis on the reference transaction data; performs secondary processing on the reference transaction data based on the cluster analysis result to obtain target transaction data. In the above manner, abnormality detection is performed on the transaction data, and secondary processing is performed on the transaction data, which can greatly improve the data quality of the transaction data. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 It is a flowchart of the first embodiment of the transaction data processing method of the present invention;
[0048] Figure 2 It is a structural block diagram of the first embodiment of the transaction data processing device of the present invention.
[0049] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0050] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0051] The embodiment of the present invention provides a transaction data processing method, device, and device-based storage medium method, referring to Figure 1 , Figure 1 The present invention is a flowchart of a first embodiment of a transaction data processing method, apparatus, and device-based storage medium method.
[0052] In this embodiment, the transaction data processing method, device, equipment and storage medium method include the following steps:
[0053] Step S10: Obtain original transaction data.
[0054] In this embodiment, the execution subject of this embodiment is a transaction data processing method, apparatus, and device-based storage medium device, wherein the transaction data processing method, apparatus, and device-based storage medium device have functions such as data processing, data communication, and program running. The transaction data processing method, apparatus, and device-based storage medium device can be a computer terminal device or other network device, and of course can also be other devices with similar functions, and this embodiment does not limit this.
[0055] It should be noted that the main method of processing transaction data at present is to collect and analyze the data before storing it. However, this method cannot improve the data quality of transaction data and there is a problem of poor data quality.
[0056] In order to solve the above technical problems, in this embodiment, the original transaction data is obtained; the original transaction data is subjected to data anomaly detection to obtain the detection result; based on the detection result, it is determined whether there is abnormal transaction data in the original transaction data, and if there is no abnormal transaction data, the original transaction data is processed once to obtain reference transaction data; the reference transaction data is subjected to cluster analysis; based on the cluster analysis result, the reference transaction data is subjected to secondary processing to obtain target transaction data. In the above manner, the transaction data is subjected to anomaly detection and secondary processing, which can greatly improve the data quality of the transaction data. Specifically, it can be achieved in the following manner.
[0057] In a specific implementation, in this embodiment, the original transaction data needs to be obtained first. The original transaction data can be obtained in real time during the transaction process, or the recorded original transaction data can be obtained after the transaction is completed.
[0058] Step S20: Perform data anomaly detection on the original transaction data to obtain a detection result.
[0059] After obtaining the original transaction data, in this embodiment, before processing the original transaction data, it is necessary to first detect whether the obtained original transaction data has anomalies. The process of performing data anomaly detection on the original transaction data is specifically to obtain a historical transaction data set, and input the features of the historical transaction data set into a preset neural network model;
[0060] Initializing the weights and biases of the learning parameters, and updating the cell states of the preset neural network model during the model training process according to the characteristics of the historical transaction data set and the learning parameters;
[0061] After completing the cell state update, a decision function is used in the output layer of the preset neural network model to determine the prediction result of the preset neural network model, and a corresponding loss function is constructed based on the prediction result; it is determined whether the loss function is within a preset range. If the loss function is not within the preset range, the weight and bias of the learning parameter are optimized to adjust the learning parameter, and the step of updating the cell state of the preset neural network model during the model training process according to the characteristics of the historical transaction data set and the learning parameter is returned to execute until the loss function is within the preset range to complete the model training and obtain the abnormal data detection model; the original transaction data is input into the abnormal data detection model to obtain the detection result.
[0062] It should be noted that in this embodiment, an abnormal data detection model is used to detect abnormal data in the original transaction data. The construction of the abnormal data detection model is as described above, wherein the cell state of the preset neural network model is also the hidden state. The specific process is to first calculate the update gate and the reset gate: the update gate determines the extent to which the old state information is passed to the new state. The reset gate determines the extent to which the new input information is combined with the old state information. The update gate determines the extent to which the new input information is combined with the old state information. t and reset gate r t It can be calculated by the following formula: t =σ(W z ·[h t-1 ,x t ]+b z ), r t =σ(W r ·[h t-1 ,x t ]+b r ), W z and b z are the weights and biases of the update gate, W r and b r is the weight and bias of the reset gate, σ is the sigmoid activation function, and h t-1is the hidden state of the previous time step, x t is the input of the current time step (corresponding to the characteristics of the historical transaction data set). Then the candidate hidden state is calculated: the candidate hidden state is the state after combining the current input and the influence of the reset gate. b1), W1 and b1 are the weights and biases of the candidate hidden states, tanh is the hyperbolic tangent activation function, r t ·h t-1 Represents the result of the old hidden state after being adjusted by the reset gate. Finally, the new hidden state is calculated: the final hidden state is the result of combining the update gate, which is a combination of the old hidden state and the new candidate hidden state. z t ·h t-1 Indicates the old state information that is retained, and Indicates the added new information. Further, the expression of the decision function in this embodiment is f(x)=sign(wx+b), where the dimension of the weight vector w is the same as the dimension of the input feature x, and each element w in w i Represents the corresponding feature x i The weight in the classification decision, b is the bias term, and the expression of the loss function is
[0063]
[0064] Among them, y is the predicted classification label corresponding to the predicted result, w and b are the weights and biases used by the learning parameters, and x i This is the input feature. The preset range can be set according to actual needs, and the value of the above loss function needs to be within the preset range.
[0065] Step S30: determining whether there is abnormal transaction data in the original transaction data based on the detection result; if there is no abnormal transaction data, processing the original transaction data once to obtain reference transaction data.
[0066] If there is abnormal transaction data, it means that there is an abnormality in the transaction. In this case, both parties to the transaction need to be reminded or the transaction operation needs to be terminated. The transaction data processed in this embodiment is the transaction data generated by normal transactions. The processing process is specifically to extract the metadata about the data attributes of the original transaction data; identify the dirty data in the metadata of the data attributes; determine the data representation of the dirty data; determine the data cleaning strategy based on the representation; clean the original transaction data based on the data cleaning strategy to obtain reference transaction data.
[0067] It should be noted that the metadata of the data attributes in this embodiment refers to information describing the data attributes (also called fields, columns or attributes), such as data attribute name: column name or field identifier, data type: such as integer, floating point number, string, date / time, Boolean value, etc., data length: the maximum length of string or binary data, etc., the manifestation of dirty data includes null values, error values and inconsistent data, so the corresponding cleaning strategies include a first cleaning strategy, a second cleaning strategy and a third cleaning strategy, the first cleaning strategy is used to clean null values, the second cleaning strategy is used to clean error values, and the third cleaning strategy is used to clean inconsistent data. Specifically, when the first cleaning strategy is used to clean null values, some missing values can be derived from this data source or other data sources, and the missing values can also be replaced by the average value, median value, maximum value, minimum value or more complex probability statistical function value, and an acceptable preset value can also be entered. When the second cleaning strategy is used to clean the error value, the possible error value or abnormal value can be identified by statistical analysis, and the error can be detected and corrected using a simple rule base (common sense rules, business-specific rules, etc.) and the error can be detected and corrected using external data. When the third cleaning strategy is used to clean inconsistent data, the format, standard, granularity and encoding method can be converted according to unified standards.
[0068] Step S40: performing cluster analysis on the reference transaction data.
[0069] In the specific implementation, this embodiment performs cluster analysis based on the clustering model, and the construction process of the clustering model is specifically as follows: starting from a SOM layer of size 2×2 in the self-organizing map network, repeatedly inputting training sample vectors into the self-organizing map network, and training the SOM layer of the self-organizing map network until a preset number of iterations is reached; detecting whether the current SOM layer satisfies the preset horizontal expansion criterion function; if the preset horizontal expansion criterion function is satisfied, then continuing to detect and check whether each neuron of the current SOM layer satisfies the preset vertical expansion criterion function; if the preset vertical expansion criterion function is satisfied, it is determined that the training of the self-organizing map network is completed to obtain the clustering model.
[0070] It should be noted that the preset horizontal expansion criterion function is MQE m <τ1·qe u ,MQE m is the average value of the quantization error of the neurons corresponding to the current SOM layer, qe u is the quantization error of the neuron u in the previous layer corresponding to the SOM graph m, τ1 is a predefined parameter value that controls the number of neurons in the SOM graph, τ1 corresponds to the number of neurons, the smaller τ1 is, the more neurons are correspondingly present. The preset vertical expansion criterion function is qei <τ2·qe0,qe i is the quantization error of neuron i in the current SOM layer, τ2 is also a predefined parameter used to control the vertical expansion of the GHSOM image. The smaller the value of τ2, the deeper the GHSOM image, and qe0 is the average value of the neuron quantization error corresponding to the initial SOM layer. If any of the above two conditions is not met, the training process is returned and the iteration continues. In addition, the preset number of iterations can be set according to actual needs. When returning to re-iteration, the number of iterations is recounted, which is not limited in this embodiment.
[0071] Step S50: performing secondary processing on the reference transaction data based on the cluster analysis result to obtain target transaction data.
[0072] Based on the clustering analysis results, in this embodiment, the reference transaction data can be processed secondary. The specific process is to determine the data classification of the reference transaction data based on the clustering analysis results; determine the data encryption level according to the data classification; determine the corresponding encryption method based on the data encryption level; encrypt the reference transaction data according to the encryption method to obtain the target transaction data.
[0073] It should be noted that cluster analysis is used to classify data. According to the clustering results, the data classification of reference transaction data can be obtained. Data classification can determine different types of transaction data. Different types of data can be encrypted in different ways. For example, the encryption of transaction record data can adopt a symmetric encryption algorithm (such as AES). For example, mobile payment data encryption can adopt SSL / TLS. The SSL / TLS protocol is used to encrypt data transmission during mobile payment and protect the payment information from being stolen during the transmission process between the client and the server. The encryption method can be selected according to actual needs, and this is not limited in this embodiment.
[0074] In this embodiment, the original transaction data is obtained; the original transaction data is subjected to data anomaly detection to obtain the detection result; based on the detection result, it is determined whether there is abnormal transaction data in the original transaction data; if there is no abnormal transaction data, the original transaction data is processed once to obtain reference transaction data; the reference transaction data is subjected to cluster analysis; based on the cluster analysis result, the reference transaction data is subjected to secondary processing to obtain target transaction data. In the above manner, the transaction data is subjected to anomaly detection and secondary processing, which can greatly improve the data quality of the transaction data.
[0075] In addition, an embodiment of the present invention further provides a storage medium, on which a transaction data processing program is stored. When the transaction data processing program is executed by a processor, the steps of the transaction data processing method described above are implemented.
[0076] Reference Figure 2 , Figure 2 This is a structural block diagram of the first embodiment of the transaction data processing method, device, equipment and storage medium device of the present invention.
[0077] like Figure 2 As shown, the transaction data processing device proposed in the embodiment of the present invention includes:
[0078] An acquisition module 10 is used to acquire original transaction data;
[0079] A detection module 20, configured to perform data anomaly detection on the original transaction data to obtain a detection result;
[0080] A processing module 30, configured to determine whether there is abnormal transaction data in the original transaction data based on the detection result, and if there is no abnormal transaction data, process the original transaction data once to obtain reference transaction data;
[0081] An analysis module 40, configured to perform cluster analysis on the reference transaction data;
[0082] The processing module 30 is used to perform secondary processing on the reference transaction data based on the cluster analysis result to obtain target transaction data.
[0083] In this embodiment, the original transaction data is obtained; the original transaction data is subjected to data anomaly detection to obtain the detection result; based on the detection result, it is determined whether there is abnormal transaction data in the original transaction data; if there is no abnormal transaction data, the original transaction data is processed once to obtain reference transaction data; the reference transaction data is subjected to cluster analysis; based on the cluster analysis result, the reference transaction data is subjected to secondary processing to obtain target transaction data. In the above manner, the transaction data is subjected to anomaly detection and secondary processing, which can greatly improve the data quality of the transaction data.
[0084] In some embodiments, the detection module 20 is used to obtain a historical transaction data set and input features of the historical transaction data set into a preset neural network model;
[0085] Initializing the weights and biases of the learning parameters, and updating the cell states of the preset neural network model during the model training process according to the characteristics of the historical transaction data set and the learning parameters;
[0086] After completing the cell state update, after completing the cell state update, using a decision function in the output layer of the preset neural network model to determine the prediction result of the preset neural network model, and constructing a corresponding loss function based on the prediction result;
[0087] Determine whether the loss function is within a preset range. If the loss function is not within the preset range, optimize the weight and bias of the learning parameter to adjust the learning parameter, and return to execute the step of updating the cell state of the preset neural network model during the model training process according to the characteristics of the historical transaction data set and the learning parameter, until the loss function is within the preset range, so as to complete the model training and obtain the abnormal data detection model;
[0088] The original transaction data is input into the abnormal data detection model to obtain a detection result.
[0089] In some embodiments, the expression of the decision function is f(x)=sign(wx+b), where the dimension of the weight vector w is the same as the dimension of the input feature x, and each element w in w i Represents the corresponding feature x i The weight in the classification decision, b is the bias term,
[0090] The expression of the loss function is Among them, y is the predicted classification label corresponding to the predicted result.
[0091] In some embodiments, the processing module 30 is used to extract metadata about data attributes of the original transaction data;
[0092] identifying dirty data in metadata of the data attributes;
[0093] Determine a data representation form of the dirty data;
[0094] Determining a data cleaning strategy based on the representation;
[0095] The original transaction data is cleaned based on the data cleaning strategy to obtain reference transaction data.
[0096] In some embodiments, the cleaning strategy includes a first cleaning strategy, a second cleaning strategy and a third cleaning strategy, wherein the first cleaning strategy is used to clean null values, the second cleaning strategy is used to clean erroneous values, and the third cleaning strategy is used to clean inconsistent data.
[0097] In some embodiments, the analysis module 40 is used to start from a SOM layer of size 2×2 in the self-organizing map network, repeatedly input training sample vectors into the self-organizing map network, and train the SOM layer of the self-organizing map network until a preset number of iterations is reached;
[0098] Detect whether the current SOM layer meets the preset horizontal expansion criterion function;
[0099] If the preset horizontal expansion criterion function is satisfied, then continue to detect and check whether each neuron in the current SOM layer satisfies the preset vertical expansion criterion function;
[0100] If the preset vertical expansion criterion function is satisfied, it is determined that the self-organizing map network training is completed to obtain a clustering model;
[0101] The reference transaction data is input into the clustering model to obtain a clustering analysis result.
[0102] In some embodiments, the processing module 30 is used to determine the data classification of the reference transaction data based on the cluster analysis result;
[0103] determining a data encryption level according to the data classification;
[0104] Determining a corresponding encryption method based on the data encryption level;
[0105] The reference transaction data is encrypted according to the encryption method to obtain target transaction data.
[0106] An embodiment of the present application also provides a transaction data processing device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus, and the memory is used to store the transaction data processing program; the processor is used to implement the above-mentioned transaction data processing method when executing the program stored in the memory.
[0107] The communication bus mentioned in the transaction data processing device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc.
[0108] The communication interface is used for communication between the above transaction data processing device and other devices.
[0109] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0110] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components.
[0111] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.
[0112] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0113] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0114] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
[0115] It should be understood that the above is only an example and does not constitute any limitation on the technical solution of the present invention. In specific applications, technicians in this field can make settings as needed, and the present invention does not limit this.
[0116] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.
[0117] In addition, for technical details not fully described in this embodiment, reference can be made to the transaction data processing method provided in any embodiment of the present invention, and will not be repeated here.
[0118] In addition, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.
[0119] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0120] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0121] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
[0122] It is understandable that the system provided by the embodiment of the present invention corresponds to the method provided by the embodiment of the present invention, and the explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the above method.
Claims
1. A transaction data processing method, characterized in that: The transaction data processing method comprises: Get raw transaction data; Performing data anomaly detection on the original transaction data to obtain a detection result; Determine whether there is abnormal transaction data in the original transaction data based on the detection result, and if there is no abnormal transaction data, process the original transaction data once to obtain reference transaction data; Performing cluster analysis on the reference transaction data; The reference transaction data is processed secondary based on the cluster analysis result to obtain target transaction data.
2. The transaction data processing method according to claim 1, characterized in that: The performing data anomaly detection on the original transaction data to obtain a detection result includes: Acquire a historical transaction data set, and input features of the historical transaction data set into a preset neural network model; Initializing the weights and biases of the learning parameters, and updating the cell states of the preset neural network model during the model training process according to the characteristics of the historical transaction data set and the learning parameters; After completing the cell state update, after completing the cell state update, using a decision function in the output layer of the preset neural network model to determine the prediction result of the preset neural network model, and constructing a corresponding loss function based on the prediction result; Determine whether the loss function is within a preset range. If the loss function is not within the preset range, optimize the weight and bias of the learning parameter to adjust the learning parameter, and return to execute the step of updating the cell state of the preset neural network model during the model training process according to the characteristics of the historical transaction data set and the learning parameter, until the loss function is within the preset range, so as to complete the model training and obtain the abnormal data detection model; The original transaction data is input into the abnormal data detection model to obtain a detection result.
3. The transaction data processing method according to claim 2, characterized in that: The expression of the decision function is f(x)=sign(wx+b), where the dimension of the weight vector w is the same as the dimension of the input feature x, and each element w in w i Represents the corresponding feature x i The weight in the classification decision, b is the bias term, and the expression of the loss function is Among them, y is the predicted classification label corresponding to the predicted result.
4. The transaction data processing method according to claim 1, characterized in that: The processing of the original transaction data to obtain reference transaction data includes: Extracting metadata about data attributes of the original transaction data; identifying dirty data in metadata of the data attributes; Determine a data representation form of the dirty data; Determining a data cleaning strategy based on the representation; The original transaction data is cleaned based on the data cleaning strategy to obtain reference transaction data.
5. The transaction data processing method according to claim 4, characterized in that: The cleaning strategy includes a first cleaning strategy, a second cleaning strategy and a third cleaning strategy, wherein the first cleaning strategy is used to clean null values, the second cleaning strategy is used to clean erroneous values, and the third cleaning strategy is used to clean inconsistent data.
6. The transaction data processing method according to claim 1, characterized in that: The clustering analysis of the reference transaction data includes: Starting from a SOM layer of size 2×2 in the self-organizing map network, repeatedly inputting training sample vectors into the self-organizing map network, and training the SOM layer of the self-organizing map network until a preset number of iterations is reached; Detect whether the current SOM layer meets the preset horizontal expansion criterion function; If the preset horizontal expansion criterion function is satisfied, then continue to detect and check whether each neuron in the current SOM layer satisfies the preset vertical expansion criterion function; If the preset vertical expansion criterion function is satisfied, it is determined that the self-organizing map network training is completed to obtain a clustering model; The reference transaction data is input into the clustering model to obtain a clustering analysis result.
7. The transaction data processing method according to any one of claims 1 to 6, characterized in that: The second processing of the reference transaction data based on the cluster analysis result to obtain the target transaction data includes: Determining data classification of the reference transaction data based on the cluster analysis result; determining a data encryption level according to the data classification; Determining a corresponding encryption method based on the data encryption level; The reference transaction data is encrypted according to the encryption method to obtain target transaction data.
8. A transaction data processing device, characterized in that: The transaction data processing device comprises: Acquisition module, used to obtain original transaction data; A detection module, used to perform data anomaly detection on the original transaction data to obtain a detection result; a processing module, configured to determine whether there is abnormal transaction data in the original transaction data based on the detection result, and if there is no abnormal transaction data, process the original transaction data once to obtain reference transaction data; An analysis module, used for performing cluster analysis on the reference transaction data; The processing module is used to perform secondary processing on the reference transaction data based on the cluster analysis result to obtain target transaction data.
9. A transaction data processing device, characterized in that: The transaction data processing device comprises: a memory, a processor, and a transaction data processing program stored in the memory and executable on the processor, wherein the transaction data processing program is configured to implement the steps of the transaction data processing method according to any one of claims 1 to 7.
10. A storage medium, characterized in that: The storage medium stores a transaction data processing program, which, when executed by a processor, implements the steps of the transaction data processing method according to any one of claims 1 to 7.