A data processing method and apparatus
By acquiring target and baseline identifiers, filtering fields, calculating binary dissimilarity and update rate, and using the mean model for anomaly marking, the problems of low efficiency and insufficient accuracy in database anomaly detection are solved, achieving efficient and accurate database anomaly detection.
Patent Information
- Application Number
- CN202310767890.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-27
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2043-06-27
AI Technical Summary
The existing database has low efficiency in detecting anomalies, high labor costs, and low flexibility in error judgment rules, resulting in a high error rate.
By acquiring target and baseline identifiers, filtering target and baseline fields, calculating binary dissimilarity and update rate, and using the mean model for anomaly labeling, the detection accuracy is improved.
It achieves efficient and accurate database anomaly detection, reduces manual identification work, and improves the accuracy of abnormal data identification.
Smart Images

Figure CN116795855B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a data processing method and apparatus. Background Technology
[0002] Currently, database anomaly detection has a very broad application prospect, providing necessary data monitoring and data alignment services to ensure the consistency and integrity of business system data and to protect the accurate synchronization of important virtual assets of enterprises.
[0003] In the process of realizing this invention, the inventors discovered at least the following problems in the prior art:
[0004] Users commonly encounter low efficiency in database anomaly detection when aligning and correcting massive amounts of data. This is because some existing technologies rely on manual checks by operations and maintenance personnel, resulting in low efficiency and high labor costs. Other existing technologies analyze real-time updated data using pre-defined rules to determine the legitimacy of data update events. While this approach employs automated data detection methods, the limited flexibility of pre-defined error-judgment rules can lead to a higher error rate. Summary of the Invention
[0005] In view of this, embodiments of the present invention provide a data processing method and apparatus that can solve the technical problem of low efficiency in existing database anomaly detection.
[0006] To achieve the above objectives, according to one aspect of the present invention, a data processing method is provided, comprising: acquiring a target identifier and a benchmark identifier; filtering to obtain multiple target fields and multiple benchmark fields corresponding to each target field; determining the benchmark field associated with each target field; for each target field: calculating a binary dissimilarity based on the associated benchmark field to obtain a first update rate, and calling a preset mean model to process and obtain a corresponding second update rate; determining the second update rate of each benchmark field and comparing it with the second update rate of the associated target field; and, in response to determining that the second update rate of a certain target field is larger than a certain value, marking the target field as an anomaly.
[0007] Optionally, the target identifier and the baseline identifier are obtained, including:
[0008] Obtain the target identifier, determine the data update time corresponding to the target identifier, and use it as the target time. Use the data update time preceding the target time as the base time, and determine the base identifier corresponding to the base time.
[0009] Optionally, the baseline field associated with each target field is determined, including:
[0010] Determine the field attributes corresponding to each target field and each baseline field;
[0011] The association corresponds to the target field and the baseline field that have the same field attributes.
[0012] Optionally, a preset mean model is invoked to process and obtain the corresponding second update rate, including:
[0013] The target field and the corresponding first update rate are input into the preset mean model;
[0014] Determine the field attribute corresponding to the target field, and select data fields in the database that correspond to the field attribute based on the first parameter value to obtain multiple comparison fields;
[0015] Calculate the Euclidean distance between the target field and each comparison field to obtain multiple field distances, and sort them.
[0016] Select the two smaller field distances from the first two parameter values, and use the distance of each selected field to the corresponding comparison field as the selected field;
[0017] Query the second update rate corresponding to each selected field, and calculate the average value by summing it with the first update rate corresponding to the target field. Use the average value as the second update rate of the target field.
[0018] Optionally, a first parameter value corresponding to the field attribute is selected from the database to obtain multiple comparison fields, including:
[0019] Multiple data fields corresponding to the stated field attribute are obtained by filtering in the database. The time difference between the data update time of each of the multiple data fields and the target time is calculated and sorted. The data field corresponding to the first few smaller time differences is selected as the comparison field.
[0020] Optionally, after marking the target field as an anomaly, the following steps are included:
[0021] Send multiple target fields marked as abnormal to the specified port;
[0022] It also receives the corresponding return data from the specified port, and updates the tagging information of the corresponding target field accordingly based on the returned data.
[0023] Optionally, after updating the tagging information of the corresponding target field based on the returned data, the process includes:
[0024] Based on the updated tagging information, the second update rate of the corresponding target field is updated accordingly;
[0025] The target field is used as training data, and the updated second update rate is used as the corresponding target value. The data is then input into the mean model. The mean model is trained so that the output of each training data point approaches the corresponding target value, thereby updating the first and second parameter values of the mean model.
[0026] In addition, the present invention also provides a data processing apparatus, including an acquisition module for acquiring a target identifier and a benchmark identifier, filtering to obtain multiple target fields and multiple benchmark fields respectively, and determining the benchmark field associated with each target field; a processing module for each target field: calculating a binary dissimilarity based on the associated benchmark field to obtain a first update rate, and calling a preset mean model to process and obtain a corresponding second update rate; and a marking module for determining the second update rate of each benchmark field and comparing it with the second update rate of the associated target field, and marking the target field as abnormal in response to determining that the second update rate of a certain target field is larger than a certain value.
[0027] One embodiment of the above invention has the following advantages or beneficial effects: The present invention obtains target identifiers and benchmark identifiers, filters out multiple target fields and multiple benchmark fields corresponding to each, determines the benchmark field associated with each target field, and identifies the target object for database anomaly detection, i.e., the target field is judged based on the benchmark field; simultaneously, the present invention calculates the binary dissimilarity of each target field based on the associated benchmark field to obtain a first update rate, and calls a preset mean model to process and obtain the corresponding second update, thus obtaining the average update rate of the target field based on the historical update rate trend of the corresponding field attribute, so as to perform more accurate update rate calculation for the target field; furthermore, the second update rate of each benchmark field is determined and compared with the second update rate of the associated target field. In response to determining that the second update rate of a certain target field is larger than a certain value, the target field is marked as abnormal. A highly flexible and timely discrimination method is used to perform anomaly discrimination of the target field, thereby improving the accuracy of the present technical solution in identifying and detecting abnormal data.
[0028] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description
[0029] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:
[0030] Figure 1 This is a schematic diagram of the main flow of the data processing method according to the first embodiment of the present invention;
[0031] Figure 2 This is a schematic diagram of the database structure according to an embodiment of the present invention;
[0032] Figure 3 This is a schematic diagram of the main flow of the data processing method according to the second embodiment of the present invention;
[0033] Figure 4 This is a schematic diagram of the main flow of the data processing method according to the third embodiment of the present invention;
[0034] Figure 5 This is a schematic diagram of the main modules of a data processing apparatus according to a first embodiment of the present invention;
[0035] Figure 6 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;
[0036] Figure 7 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation
[0037] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0038] Figure 1 This is a schematic diagram of the main flow of the data processing method according to the first embodiment of the present invention, as shown below. Figure 1 As shown, the data processing method includes:
[0039] Step S101: Obtain the target identifier and the benchmark identifier, filter to obtain multiple target fields and multiple benchmark fields respectively, and determine the benchmark field associated with each target field.
[0040] In this embodiment, this step achieves the technical objective of identifying database anomaly detection objects. Specifically, the target field corresponding to the target identifier is the database anomaly detection object, and the benchmark field corresponding to the benchmark identifier is used as the corresponding anomaly detection evaluation standard. In a further embodiment, the benchmark field can be a verified and correctly updated database data field. Therefore, for each target field to be evaluated, the corresponding evaluation standard is highly flexible and timely, rather than a pre-set fixed rule, thereby increasing the accuracy of anomaly detection for the data field to be tested.
[0041] In some embodiments, to increase the efficiency of identifying abnormal data in the database, when obtaining the target identifier and the baseline identifier, the target identifier can be obtained, and the data update time corresponding to the target identifier can be determined as the target time. The data update time preceding the target time can be used as the baseline time, and the baseline identifier corresponding to the baseline time can be determined. In corresponding embodiments, it can be assumed that, apart from the target time, all database data updated in previous times have completed the corresponding data verification and correction in a timely manner. Therefore, directly setting the above-mentioned baseline time as the data update time preceding the target time by default can enable more accurate identification of abnormal data in the target field based on the most recently updated database data field, avoiding the situation where the data update offset is large and the data abnormality identification error rate is high due to the long interval between update cycles.
[0042] In some embodiments, to more accurately compare and distinguish each target field, when determining the reference field associated with each target field, the field attributes corresponding to each target field and each reference field can be determined respectively; target fields and reference fields corresponding to the same field attributes can be associated. For example, the database data storage structure is as follows: Figure 2 As shown in the figure, different data tables correspond to different data update times, and different fields in the same data table correspond to different field attributes. Therefore, analyzing and processing the updated data (i.e., the target field) based on historical data (i.e., the baseline field) corresponding to the same field attribute can make the corresponding processing results more targeted and convincing, thereby improving the data processing efficiency of this invention.
[0043] Step S102: For each target field: calculate the binary dissimilarity based on the associated baseline field to obtain the first update rate, and call the preset mean model to process and obtain the corresponding second update rate.
[0044] In this embodiment, after this step, the first update rate of the target field compared to the baseline field can be obtained, that is, the percentage of the updated part in the target field. Then, by calling the mean model, the first update rate of the target field corresponding to the same field attribute is compared with the second update rate of multiple comparison fields. Based on the overall change trend of the field attribute in the database, the mean update rate of the target field (i.e., the second update rate) is calculated, thereby obtaining the data source for anomaly judgment of the target field. Through multiple analyses, comparisons and calculations of the data dimension and update rate dimension of the target field, the accuracy of the database anomaly detection of the present invention can be effectively improved.
[0045] For example, for field attribute A, if the base field is known to be X = {1,0,0,0,1,0,1,1} and the target field is known to be Y = {0,0,0,1,1,1,1,1}, then by comparison, we can find that the 2nd, 3rd, 5th, 7th and 8th elements of field Y are the same as those of field X, while the 1st, 4th and 6th elements are different from those of field X. Therefore, the corresponding binary dissimilarity is calculated to be 3 / 8 = 0.375, which is the first update rate corresponding to the target field Y.
[0046] In some embodiments, to update the update rate of each target field to be tested based on multiple corresponding historical data in the database, thereby enabling more accurate anomaly detection of the target field, a preset mean model can be invoked to process and obtain the corresponding second update rate. This includes: inputting the target field and the corresponding first update rate into the preset mean model; determining the field attribute corresponding to the target field, and selecting data fields corresponding to the field attribute in the database with a first parameter value to obtain multiple comparison fields; calculating the Euclidean distance between the target field and each comparison field to obtain multiple field distances and sorting them; selecting the first few smaller field distances with a second parameter value, and using the corresponding comparison field of each selected field distance as the selected field; querying the second update rate corresponding to each selected field, and averaging it with the first update rate corresponding to the target field, and using the average value as the second update rate of the target field. It is worth noting that the first parameter value and the second parameter value are both parameters included in the mean model, and the first parameter value must be greater than or equal to the second parameter value. Through the mean model processing in this step, multiple historical data fields (i.e., comparison bullets) corresponding to the same field attribute can be selected from the database. Then, using the nearest neighbor algorithm, multiple target data fields with the smallest difference from the data field at the time of test are selected from the multiple data fields. The arithmetic mean of the second update rate of the multiple target data fields and the first update rate of the data field at the time of test is calculated to obtain the second update rate corresponding to the data field at the time of test. This second update rate is used as a representation of the overall distance between the target field and multiple comparison fields with the same field attribute. Thus, by measuring the distance between the target field with unknown accuracy and the default accurate comparison field, it is possible to judge whether the target field is abnormal data in the database, thereby achieving efficient utilization of historical data.
[0047] In some embodiments, to avoid a large data offset due to a long update interval, which could lead to a high misjudgment rate for the target field, a first parameter value of data fields corresponding to the field attribute can be selected from the database to obtain multiple comparison fields. In this case, multiple data fields corresponding to the field attribute can be filtered from the database, the time difference between the data update time of each of the multiple data fields and the target time can be calculated, and the data fields with the smallest first parameter value of time difference can be selected as comparison fields.
[0048] Step S103: Determine the second update rate of each baseline field and compare it with the second update rate of the associated target field. In response to determining that the second update rate of a certain target field is larger than a certain value, mark the target field as an anomaly.
[0049] In this embodiment, by comparing the second update rate of the target field with the second update rate of the benchmark field, a comparison of the data update range of the target field and the benchmark field is completed. Since the benchmark field has been verified and corrected, it can be determined to be normal data. Therefore, when the data update range of the target field is small, it can be considered that the data update action space corresponding to the target field is within the normal threshold, and thus it is determined that the updated target field will not have any anomalies. However, when the data update range of the target field is large, it is determined that the data update action space of the target time is too large, and there is a suspected data anomaly event. It needs to be marked and sent to the downstream processing end for re-verification and judgment. Through this step, the corresponding iteration of the reasonable data update action space can be completed through the iteration of the benchmark field, thereby enabling efficient and accurate automatic identification of abnormal data and saving a lot of manpower identification work.
[0050] In some embodiments, to make a final determination and correction of target fields suspected of having data anomalies, multiple marked target fields can be sent to a designated port after the target field is marked as abnormal; and the corresponding return data from the designated port can be received to update the marking information of the corresponding target fields accordingly. After this step, the suspected abnormal data fields can be aggregated to the manual processing port, thereby verifying and correcting the suspected abnormal data, obtaining the corresponding correct data, and recording it in the database as the benchmark field for judging the target fields at the next data update time.
[0051] In some embodiments, in order to continuously optimize the mean model and improve the calculation accuracy of the subsequent mean model, the labeling information of the corresponding target field can be updated according to the returned data, and then the second update rate of the corresponding target field can be updated according to the updated labeling information. The target field is used as training data, the updated second update rate is used as the corresponding target value, and input into the mean model. The mean model is trained with the goal of the output corresponding to each training data approaching the corresponding target value, thereby updating the first parameter value and the second parameter value of the mean model. For example, if for field attribute A, the second update rate of the corresponding baseline field is 0.3, and the mean model initially calculates the second update rate of the target field to be 0.4, since 0.4 > 0.3, the target field is marked as abnormal and uploaded to the designated port; the return information from the designated port is received, and the marking information of the data field at the time to be tested is updated to normal according to the return information. Since the reasonable range of the second update rate of the data field at the time to be tested is {0, 0.3}, the data field at the time to be tested is used as training data, and {0, 0.3} is the corresponding target value, to further optimize and update the mean model.
[0052] Figure 3 This is a schematic diagram of the main flow of a data processing method according to a second embodiment of the present invention, the data processing method including:
[0053] Step S301: Obtain the time to be measured and determine the standard time.
[0054] Step S302: Filter the database to obtain data corresponding to the time to be tested and the standard time, including multiple data fields corresponding to multiple field attributes.
[0055] Step S303: For each field attribute, based on the corresponding data field at the standard time, perform binary dissimilarity calculation on the corresponding data field at the time to be tested to obtain the corresponding first update rate.
[0056] Step S304: Input the data field and corresponding first update rate of each time point to be tested into the preset mean model.
[0057] Step S305: By selecting data fields in the database that correspond to the same field attribute as the first parameter value, multiple comparison data fields are obtained.
[0058] Step S306: Calculate the Euclidean distance between each comparison data field and the data field to be tested, and obtain the distance between multiple data fields.
[0059] Step S307: Sort the distances of multiple data fields and select the data field with the smaller second parameter value. Determine the distance of each selected data field to the corresponding comparison data field as the target data field.
[0060] Step S308: Query the second update rate corresponding to each target data field, and accumulate it with the first update rate corresponding to the data field at the time to be tested. Calculate the arithmetic mean of the accumulated results to serve as the second update rate corresponding to the data field at the time to be tested, and output it.
[0061] Step S309: Query the second update rate corresponding to each standard time data field and compare it with the second update rate of the corresponding test time data field.
[0062] Step S310: In response to determining that the second update rate corresponding to a certain data field at a certain time to be tested is a large value, an anomaly mark is made on the data field at the time to be tested.
[0063] Step S311: Summarize multiple abnormal data fields and upload them to the specified port.
[0064] Step S312: Receive the corresponding return data from the specified port, update the tag information corresponding to each abnormal data field according to the return data, and update the corresponding second update rate of each abnormal data field according to the updated tag information.
[0065] Step S313: Optimize the mean model based on the updated abnormal data field and its second update rate.
[0066] Figure 4 This is a schematic diagram of the main flow of a data processing method according to a third embodiment of the present invention, the data processing method including:
[0067] Step S401: Obtain the target identifier.
[0068] Step S402: Determine the data update time corresponding to the target identifier as the target time, take the previous data update time of the target time as the reference time, and determine the corresponding reference identifier.
[0069] Step S403: Determine the field attributes corresponding to each target field and each baseline field.
[0070] Step S404: Associate the target field and the baseline field that correspond to the same field attribute.
[0071] Step S405: For each target field: calculate the binary dissimilarity based on the associated baseline field to obtain the first update rate.
[0072] Step S406: Call the preset mean model to process and obtain the second update rate corresponding to each first update rate.
[0073] Step S407: Determine the second update rate for each baseline field and compare it with the second update rate of the associated target field.
[0074] In step S408, in response to determining that the second update rate of a certain target field is a large value, the target field is marked as abnormal.
[0075] Step S409: Send multiple target fields marked as abnormal to the specified port.
[0076] Step S410: Receive the corresponding return data from the specified port, and update the tag information of the corresponding target field accordingly based on the return data.
[0077] Step S411: Update the second update rate of the corresponding target field according to the updated tag information.
[0078] Step S412: Use the updated second update rate and the corresponding target field to train and optimize the second model.
[0079] Figure 5 This is a schematic diagram of the main modules of a data processing apparatus according to an embodiment of the present invention, such as... Figure 5 As shown, the data processing device 500 includes an acquisition module 501, a processing module 502, and a marking module 503. The acquisition module 501 acquires target identifiers and baseline identifiers, filters to obtain multiple target fields and multiple baseline fields corresponding to each target field, and determines the baseline field associated with each target field. The processing module 502 calculates the binary dissimilarity of each target field based on the associated baseline field to obtain a first update rate, and calls a preset mean model to process and obtain a corresponding second update rate. The marking module 503 determines the second update rate of each baseline field and compares it with the second update rate of the associated target field. In response to determining that the second update rate of a target field is larger than a certain value, the target field is marked as an anomaly.
[0080] In some embodiments, the acquisition module 501 is configured to: acquire the target identifier when acquiring the target identifier and the reference identifier, determine the data update time corresponding to the target identifier as the target time, take the previous data update time of the target time as the reference time, and determine the reference identifier corresponding to the reference time.
[0081] In some embodiments, the acquisition module 501 is configured to: determine the field attributes corresponding to each target field and each benchmark field when determining the benchmark field associated with each target field; and associate the target field and the benchmark field that correspond to the same field attributes.
[0082] In some embodiments, the processing module 502 is configured to: input the target field and the corresponding first update rate into the preset mean model when calling a preset mean model to process and obtain the corresponding second update rate; determine the field attribute corresponding to the target field, and select a first parameter value of data fields corresponding to the field attribute in the database to obtain a plurality of comparison fields; calculate the Euclidean distance between the target field and each comparison field to obtain a plurality of field distances and sort them; select the first two smaller field distances and use the corresponding comparison field of each selected field distance as the selected field; query the second update rate corresponding to each selected field, and accumulate it with the first update rate corresponding to the target field to calculate the average value, and use the average value as the second update rate of the target field.
[0083] In some embodiments, the processing module 502 is configured to: select a first parameter value of data fields corresponding to the field attribute in the database to obtain multiple comparison fields; filter multiple data fields corresponding to the field attribute in the database; calculate the time difference between the data update time corresponding to the multiple data fields and the target time; sort them; and select the data field corresponding to the first parameter value of smaller time difference as the comparison field.
[0084] In some embodiments, the marking module 503 is configured to: after marking the target field as abnormal, send multiple target fields marked as abnormal to a specified port; and receive the corresponding return data from the specified port, so as to update the marking information of the corresponding target field according to the return data.
[0085] In some embodiments, the processing module 502 is configured to: update the labeling information of the corresponding target field according to the returned data, and then update the second update rate of the corresponding target field according to the updated labeling information; use the target field as training data, use the updated second update rate as the corresponding target value, and input it into the mean model, train the mean model with the goal of the output corresponding to each training data approaching the corresponding target value, thereby updating the first parameter value and the second parameter value of the mean model.
[0086] It should be noted that the data processing method and the data processing device described in this invention have a corresponding relationship in their specific implementation, so repeated content will not be described again.
[0087] Figure 6 An exemplary system architecture 600 is shown that can be applied to the data processing method or data processing apparatus of the present invention.
[0088] like Figure 6As shown, system architecture 600 may include terminal devices 601, 602, and 603, a network 604, and a server 605. Network 604 serves as the medium for providing communication links between terminal devices 601, 602, and 603 and server 605. Network 604 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.
[0089] Users can use terminal devices 601, 602, and 603 to interact with server 605 via network 604 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 601, 602, and 603.
[0090] Terminal devices 601, 602, and 603 can be various electronic devices with a page display processing screen that supports web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.
[0091] Server 605 can be a server that provides various services, such as a backend management server that supports users using terminal devices 601, 602, and 603 (for example only). The backend management server can analyze and process data such as received product information query requests, and feed back the processing results (such as target push information, product information - for example only) to the terminal devices.
[0092] It should be noted that the data processing method provided in the embodiments of the present invention is generally executed by server 605, and correspondingly, the computing device is generally located in server 605.
[0093] It should be understood that Figure 6 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.
[0094] The following is for reference. Figure 7 It shows a schematic diagram of the structure of a computer system 700 suitable for implementing a terminal device of the present invention. Figure 7 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.
[0095] like Figure 7As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 702 or programs loaded from storage section 708 into random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the computer system 700. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0096] The following components are connected to I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display processor (LCD), speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to I / O interface 707 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 710 as needed so that computer programs read from it can be installed into storage section 708 as needed.
[0097] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by central processing unit (CPU) 701, it performs the functions defined above in the system of this invention.
[0098] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.
[0099] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0100] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including an acquisition module, a processing module, and a marking module. The names of these modules do not necessarily limit the module itself.
[0101] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to: acquire a target identifier and a reference identifier; filter to obtain multiple target fields and multiple reference fields corresponding to each target field; determine the reference field associated with each target field; for each target field: calculate a binary dissimilarity based on the associated reference field to obtain a first update rate, and call a preset mean model to process and obtain a corresponding second update rate; determine the second update rate of each reference field and compare it with the second update rate of the associated target field; in response to determining that the second update rate of a certain target field is a larger value, mark the target field as an anomaly.
[0102] The technical solution of the present invention can solve the technical problem of low efficiency in existing database anomaly detection.
[0103] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A data processing method, characterized in that, include: Obtain the target identifier and the benchmark identifier, filter to obtain multiple target fields and multiple benchmark fields respectively, and determine the benchmark field associated with each target field; For each target field: calculate the binary dissimilarity based on the associated benchmark field to obtain the first update rate, including: obtaining the number of different fields by comparing the benchmark field and the target field, and taking the percentage of the number of different fields to the total number of target fields as the first update rate; wherein, each field includes multiple elements; The process involves calling a preset mean model to process and obtain the corresponding second update rate, including: inputting the target field and the corresponding first update rate into the preset mean model; determining the field attribute corresponding to the target field, selecting data fields corresponding to the field attribute from the database with a first parameter value, and obtaining multiple comparison fields; calculating the Euclidean distance between the target field and each comparison field, obtaining multiple field distances, and sorting them in ascending order according to the field distances; selecting the first few smaller field distances with a second parameter value, and using the corresponding comparison field of each selected field distance as the selected field; querying the second update rate corresponding to each selected field, averaging it with the first update rate corresponding to the target field, and using the average value as the second update rate of the target field; A second update rate is determined for each baseline field and compared with the second update rate of the associated target field. In response to determining that the second update rate of a target field is larger than a certain value, the target field is marked as abnormal. Specifically, when the data update range of the target field is small, it is confirmed that the data update action space corresponding to the target field is within a normal threshold; when the data update range of the target field is large, it is confirmed that the data update action space corresponding to the target field is abnormal. The data update action space refers to the numerical range of data updates, and the data update range refers to the amount of data updated. The second update rate of the baseline field refers to the average update rate of the baseline field. The larger value refers to the greater of the second update rate of the baseline field and the second update rate of the target field. A smaller data update range for the target field means that the second update rate of the target field is less than the second update rate of the baseline field; a larger data update range for the target field means that the second update rate of the target field is greater than the second update rate of the baseline field.
2. The method according to claim 1, characterized in that, Obtain the target identifier and the baseline identifier, including: Obtain the target identifier, determine the data update time corresponding to the target identifier, and use it as the target time. Use the data update time preceding the target time as the base time, and determine the base identifier corresponding to the base time.
3. The method according to claim 1, characterized in that, Determine the baseline field associated with each target field, including: Determine the field attributes corresponding to each target field and each baseline field; The association corresponds to the target field and the baseline field that have the same field attributes.
4. The method according to claim 2, characterized in that, Select the first parameter value from the database corresponding to the field attribute to obtain multiple comparison fields, including: Multiple data fields corresponding to the stated field attribute are obtained by filtering in the database. The time difference between the data update time of each of the multiple data fields and the target time is calculated. The data fields corresponding to the first few smaller time differences are selected as comparison fields.
5. The method according to claim 1, characterized in that, After marking the target field as an anomaly, the following steps are taken: Send multiple target fields marked as abnormal to the specified port; It also receives the corresponding return data from the specified port, and updates the tagging information of the corresponding target field accordingly based on the returned data.
6. The method according to claim 5, characterized in that, After updating the tagging information of the corresponding target fields based on the returned data, the following is included: Based on the updated tagging information, the second update rate of the corresponding target field is updated accordingly; The target field is used as training data, and the updated second update rate is used as the corresponding target value. The data is then input into the mean model. The mean model is trained so that the output of each training data point approaches the corresponding target value, thereby updating the first and second parameter values of the mean model.
7. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire target identifiers and benchmark identifiers, filter them to obtain multiple target fields and multiple benchmark fields respectively, and determine the benchmark field associated with each target field; The processing module is used to: calculate the binary dissimilarity based on the associated benchmark field for each target field to obtain a first update rate, including: comparing the benchmark field and the target field to obtain the number of different fields, and taking the percentage of the number of different fields to the total number of target fields as the first update rate; wherein each field includes multiple elements; and call a preset mean model to process and obtain the corresponding second update rate, including: inputting the target field and the corresponding first update rate into the preset mean model; determining the field attribute corresponding to the target field, and selecting a first parameter value of data fields corresponding to the field attribute in the database to obtain multiple comparison fields; calculating the Euclidean distance between the target field and each comparison field to obtain multiple field distances, and sorting them in ascending order according to the field distances; selecting the first two smaller field distances, and taking the corresponding comparison field of each selected field distance as the selected field; querying the second update rate corresponding to each selected field, and averaging it with the first update rate corresponding to the target field, and taking the average as the second update rate of the target field; A marking module is used to determine the second update rate of each baseline field and compare it with the second update rate of the associated target field. In response to determining that the second update rate of a target field is larger than a certain value, the target field is marked as abnormal. Specifically, when the data update range of the target field is small, it is confirmed that the data update action space corresponding to the target field is within a normal threshold; when the data update range of the target field is large, it is confirmed that the data update action space corresponding to the target field is abnormal. The data update action space refers to the numerical range of data updates, and the data update range refers to the amount of data updated. The second update rate of the baseline field refers to the average update rate of the baseline field. The larger value refers to the greater of the second update rate of the baseline field and the second update rate of the target field. A smaller data update range for the target field means that the second update rate of the target field is less than the second update rate of the baseline field; a larger data update range for the target field means that the second update rate of the target field is greater than the second update rate of the baseline field.
8. The apparatus according to claim 7, characterized in that, Obtain the target identifier and the baseline identifier, including: Obtain the target identifier, determine the data update time corresponding to the target identifier, and use it as the target time. Use the data update time preceding the target time as the base time, and determine the base identifier corresponding to the base time.
9. The apparatus according to claim 7, characterized in that, Determine the baseline field associated with each target field, including: Determine the field attributes corresponding to each target field and each baseline field; The association corresponds to the target field and the baseline field that have the same field attributes.
10. The apparatus according to claim 8, characterized in that, Select the first parameter value from the database corresponding to the field attribute to obtain multiple comparison fields, including: Multiple data fields corresponding to the stated field attribute are obtained by filtering in the database. The time difference between the data update time and the target time corresponding to each of the multiple data fields is calculated. The data fields corresponding to the first few smaller time differences are selected as comparison fields.
11. The apparatus according to claim 7, characterized in that, After marking the target field as an anomaly, the following steps are taken: Send multiple target fields marked as abnormal to the specified port; It also receives the corresponding return data from the specified port, and updates the tagging information of the corresponding target field accordingly based on the returned data.
12. The apparatus according to claim 11, characterized in that, After updating the tagging information of the corresponding target fields based on the returned data, the following is included: Based on the updated tagging information, the second update rate of the corresponding target field is updated accordingly; The target field is used as training data, and the updated second update rate is used as the corresponding target value. The data is then input into the mean model. The mean model is trained so that the output of each training data point approaches the corresponding target value, thereby updating the first and second parameter values of the mean model.
13. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.
14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.
15. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.
Citation Information
Patent Citations
Database enhancement method and device
CN110008209A
Bill data processing method and device, electronic equipment and storage medium
CN115391343A
