Database loading exception handling method, apparatus, device, and medium

Abstract

The application provides a database loading exception processing method, device, equipment and medium, which can be applied to the field of electric digital data processing. The method comprises the following steps: executing a loading task for to-be-loaded data; in response to triggering an exception event during the execution of the loading task, determining an exception feedback mode according to the data amount of the to-be-loaded data remaining in the loading task, wherein the exception feedback mode indicates that an exception attribute field of the exception event is sent to a loading tool, or the event statistical quantity of the exception event is counted and the current event statistical quantity is fed back to the loading tool under the condition that a sub-loading task for a specified storage area is completed; and in the process of executing the loading task, in response to a target instruction from the loading tool, stopping the loading task, wherein the loading tool determines a target statistical quantity according to at least one of the exception attribute field and the event statistical quantity, and sends the target instruction to the database end under the condition that the target statistical quantity meets a preset exception event quantity condition.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing, and more specifically to a method, apparatus, device, and medium for handling database loading anomalies. Background Technology

[0002] Database data loading is a routine operation in applications, widely used in scenarios such as data migration and backup / recovery. Exceptions may occur during loading; when an exception occurs, the database will generate an error and interrupt the loading process, and already loaded data will be rolled back and become invisible. During large-scale data loading, the loading time is relatively long. How to efficiently record abnormal data while ensuring the successful loading of correct data is a pressing problem that needs to be solved. Summary of the Invention

[0003] In view of the above problems, the present invention provides a database loading exception handling method, apparatus, device and medium.

[0004] According to a first aspect of the present invention, a database loading exception handling method is provided, comprising: executing a loading task for data to be loaded; in response to an exception event triggered by executing the loading task, determining an exception feedback method based on the amount of remaining data to be loaded in the loading task, wherein the exception feedback method indicates sending the exception attribute field of the exception event to a loading tool, or counting the event statistics of the exception event, and feeding back the current event statistics to the loading tool after completing a sub-loading task for a specified storage area; during the execution of the loading task, stopping the loading task in response to a target instruction from the loading tool, wherein the loading tool determines a target statistical number based on at least one of the exception attribute field and the event statistical number, and sends a target instruction to the database if the target statistical number meets a preset exception event number condition.

[0005] Optionally, the exception feedback method is determined based on the amount of remaining unloaded data in the loading task, including: if the amount of remaining unloaded data in the loading task is greater than a preset data amount threshold, determining the first parameter item as the first feedback value; performing exception event statistics on the detected exception events based on the first feedback value to obtain the current event statistics count; and, if the sub-loading task for the specified storage area is completed, reporting the currently obtained event statistics count to the loading tool.

[0006] Optionally, the exception feedback method is determined based on the amount of data remaining to be loaded in the loading task. This also includes: recording the exception information of the exception event in the database cache according to the first feedback value; and, when the sub-loading task for the specified storage area is completed, performing item statistics on multiple exception information corresponding to the specified storage area to obtain the current event statistics count, and feeding back the currently obtained event statistics count to the loading tool.

[0007] Optionally, the method of determining the exception feedback based on the amount of data remaining to be loaded in the loading task may further include: if the amount of data remaining to be loaded in the loading task is less than or equal to a preset data amount threshold, determining the parameter value of the first parameter item as the second feedback value; and feeding back the exception attribute field of the exception event to the loading tool based on the second feedback value.

[0008] Optionally, the target statistical quantity can be determined based on at least one of the exception attribute field and the event statistics quantity, including: summing the event statistics quantities corresponding to multiple specified storage areas to obtain the area statistics quantity; and determining the area statistics quantity as the target statistical quantity.

[0009] Optionally, the target statistical quantity is determined based on at least one of the abnormal attribute field and the event statistical quantity, and further includes: in response to the abnormal attribute field of the received abnormal event, performing an auto-incrementing statistical count based on the statistical plugin on the loading tool to determine the single abnormal statistical quantity; and determining the target statistical quantity based on the sum of the district statistical quantity and the single abnormal statistical quantity.

[0010] Optionally, the exception attribute fields include an exception attribute name field and a row number field of the specified storage area where the exception event is located. It also includes: if the first parameter value is the first feedback value, determining the original data from the data to be loaded based on the exception information recorded in the cache in the database; or if the first parameter value is the second feedback value, calling the loading tool to determine the original data from the specified storage area based on the row number field and the exception attribute name field; and analyzing the original data to obtain a processing strategy.

[0011] A second aspect of the present invention provides a database loading exception handling apparatus, comprising: an execution module for executing a loading task for data to be loaded; a response module for responding to an exception event triggered by executing the loading task, determining an exception feedback method based on the amount of remaining data to be loaded in the loading task, wherein the exception feedback method includes sending the exception attribute field of the exception event to the loading tool, or counting the event statistics of the exception event, and feeding back the current event statistics to the loading tool after completing a sub-loading task for a specified storage area; and a stop module for stopping the loading task in response to a target instruction from the loading tool during the execution of the loading task, wherein the loading tool determines a target statistical number based on at least one of the exception attribute field and the event statistical number, and sends a target instruction to the database if the target statistical number meets a preset exception event number condition.

[0012] A third aspect of the present invention provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the method described above.

[0013] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0014] According to the database loading exception handling method, apparatus, device, and medium provided by the present invention, the exception feedback method of different granularities can be automatically adjusted according to the amount of remaining data to be loaded in the database loading task. When there is a lot of remaining data to be loaded, that is, in the early stage of the loading task, the database performs coarse-grained event counting on a storage area basis, and the loading tool directly merges the event counting from multiple storage areas, improving the efficiency of the loading tool in determining the target counting quantity. When there is a little remaining data to be loaded, that is, in the later stage of the loading task, the database does not record exception information, but feeds back exception attribute fields to the loading tool in real time. The loading tool performs fine-grained counting on a single exception event basis, improving the precision of the calculation of the number of triggered target instructions and reducing computational costs. This approach avoids resource waste by allowing the database to continue loading without handling exceptions or rolling back when abnormal events are triggered. It improves the efficiency of loading normal database data, avoids the database table bloat caused by rollback, and thus does not affect the performance of subsequent table operations. The flexible adjustment of the two exception feedback methods reduces the complexity of filtering data during exceptions, improving the success rate of data loading and operational efficiency. Furthermore, since the target statistical number generated based on data anomalies generally does not meet the preset exception event quantity condition, it is insufficient to trigger the loading tool to issue a stop-loading instruction, and the database will continue loading. However, a large number of exceptions caused by loading tool issues such as coding problems will cause the loading tool and database to quickly exit the loading process, reducing resource waste and optimizing the exception handling process. Attached Figure Description

[0015] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings.

[0016] Figure 1 The following is an application scenario of the database loading exception handling method and apparatus according to an embodiment of the present invention.

[0017] Figure 2 A flowchart of a database loading exception handling method according to an embodiment of the present invention is shown.

[0018] Figure 3A flowchart illustrating the database-side loading task execution according to an embodiment of the present invention is shown.

[0019] Figure 4 A flowchart illustrating the execution of a loading tool according to an embodiment of the present invention is shown.

[0020] Figure 5 A structural block diagram of a database loading exception handling apparatus according to an embodiment of the present invention is shown.

[0021] Figure 6 A block diagram of an electronic device suitable for implementing a database loading exception handling method according to an embodiment of the present invention is shown. Detailed Implementation

[0022] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0023] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0024] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0025] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0026] In implementing this invention, it was discovered that when a database fails to load, it rolls back, and all successfully loaded data becomes invisible. This characteristic is known as the atomicity of a database. Atomicity means that all operations within a transaction either succeed completely or fail completely and roll back. Therefore, if a transaction's operations succeed, they must be fully applied to the database; if they fail, they should have no impact on the database. Based on the atomicity of the database, exception handling during data loading is generally initiated by the loading tool. This involves introducing an exception handling task to process the data in the failure buffer, splitting the buffer into strips or shards for loading, continuously rolling back to filter out abnormal data, and finally completing the filtering of abnormal data and loading of normal data from the buffer. However, this approach increases unnecessary network interactions, CPU processing time, and storage space, significantly impacting the efficiency of subsequent table operations in the database.

[0027] In view of this, embodiments of the present invention provide a database loading exception handling method, apparatus, device, and medium. The method includes: executing a loading task for data to be loaded; in response to an exception event triggered by executing the loading task, determining an exception feedback method based on the amount of remaining data to be loaded in the loading task, wherein the exception feedback method indicates sending the exception attribute field of the exception event to a loading tool, or counting the event statistics of the exception event, and, upon completion of a sub-loading task for a specified storage area, reporting the current event statistics to the loading tool; during the execution of the loading task, stopping the loading task in response to a target instruction from the loading tool, wherein the loading tool determines a target statistical number based on at least one of the exception attribute field and the event statistical number, and, if the target statistical number meets a preset exception event number condition, sending a target instruction to the database.

[0028] In the technical solution of this invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0029] Figure 1 The following is an application scenario of the database loading exception handling method and apparatus according to an embodiment of the present invention.

[0030] like Figure 1As shown, the business system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0031] Users can interact with server 105 via network 104 using at least one of the first terminal device 101, second terminal device 102, and third terminal device 103 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, second terminal device 102, and third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0032] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0033] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The clustered distributed database is deployed on the clustered server 105. Server 105 includes multiple server nodes that can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0034] It should be noted that the database loading exception handling method provided in this embodiment of the invention can generally be executed by server 105. Correspondingly, the database loading exception handling device provided in this embodiment of the invention can generally be located in server 105. The database loading exception handling method provided in this embodiment of the invention can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the database loading exception handling device provided in this embodiment of the invention can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0035] Alternatively, the database loading exception handling method provided in this embodiment of the invention can also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Correspondingly, the database loading exception handling device provided in this embodiment of the invention can also be located in the first terminal device 101, the second terminal device 102, or the third terminal device 103, or in other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0036] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0037] It should be noted that the sequence numbers of the operations in the following methods are for descriptive purposes only and should not be considered as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed in the exact order shown.

[0038] Figure 2 A flowchart of a database loading exception handling method according to an embodiment of the present invention is shown.

[0039] like Figure 2 As shown, the method 200 includes operations S210 to S230.

[0040] In operation S210, a loading task for the data to be loaded is executed.

[0041] Optionally, the database loading exception handling method is applied to the database side, which is the database server.

[0042] Optionally, a loading tool is launched. The loading tool includes multiple storage areas, which can be buffers. The loading tool uses a multi-loading task batch processing method to load data. The data to be loaded from the target loading table is first pulled into multiple storage areas. For example, each storage area contains multiple records of 100M in size.

[0043] Optionally, the loading tool starts multiple threads to establish connections with the database. Each thread is responsible for packaging the data to be loaded in its corresponding storage area and sending it to the database. The database server executes the loading task for the data to be loaded, that is, parsing each record in the data to be loaded one by one, and storing the successfully parsed records in an empty table in the database. The metadata of the empty table is the same as that of the target loading table.

[0044] Optionally, multiple threads can load concurrently, implementing a concurrent batch loading method, which makes the entire loading process highly efficient.

[0045] In operation S220, in response to an exception event triggered by the execution of a loading task, the exception feedback method is determined based on the amount of data remaining to be loaded in the loading task.

[0046] Optional exceptions include data field values ​​that do not match table field types, violations of uniqueness constraints, etc.

[0047] Optionally, the exception feedback method can be to send the exception attribute field of the exception event to the loading tool, or to count the number of exception events and, upon completion of the sub-loading task for the specified storage area, report the current number of events to the loading tool.

[0048] Optionally, before execution on the database side, the loading tool inputs the loading options related to the newly added exception feedback method and the data to be loaded into the database through the connection established with the database, and the database side begins to execute the loading task.

[0049] Optionally, a coarse-grained exception feedback method can be used, representing the event count of exception events according to the storage area. That is, the database counts the event count of exception events and reports the current event count to the loading tool after completing the sub-loading task for the specified storage area. In this case, the database can cache the local exception information.

[0050] For example, the data to be loaded in storage area A is sent to the database. The database executes the loading task for storage area A. In response to an abnormal event, the database increments the event count of the abnormal event in storage area A by 1 until the database completes the loading task in storage area A and then reports the current event count to the loading tool.

[0051] Optionally, a real-time feedback method can be adopted that represents the anomaly at a fine-grained level, based on individual anomaly events. In this case, the database does not cache local anomaly information and directly sends the anomaly attribute fields of the anomaly events to the loading tool.

[0052] For example, data to be loaded in storage area A is sent to the database. The database executes a loading task for storage area A. In response to an abnormal event, the database sends the abnormal attribute field of the abnormal event back to the loading tool in real time. The loading tool increments the count of abnormal events by 1.

[0053] Optionally, in response to an exception event triggered during the execution of the loading task, the database side continues to execute the loading task and proceeds to parse the next record.

[0054] During operation of S230, in response to the target instruction from the loading tool, the loading task is stopped while the loading task is being executed.

[0055] Optionally, when using a coarse-grained exception feedback method, the loading tool accumulates the event statistics from multiple storage areas to reach a target statistical number. If the target statistical number meets the preset exception event count condition, the tool sends a target instruction to the database.

[0056] Optionally, when using a fine-grained exception feedback method, the loading tool counts the cumulative number of single exception events based on the exception attribute fields received in real time, obtains the target number of statistics based on the sum of the cumulative number of single exception events and the number of events received in historical time periods, and sends the target instruction to the database if the target number of statistics meets the preset exception event quantity condition.

[0057] Optionally, the target statistics represent the cumulative number of abnormal events counted on the loading tool side.

[0058] Optionally, the target statistical quantity meeting the preset abnormal event quantity condition means that the target statistical quantity is greater than the preset quantity threshold set in the preset abnormal event quantity condition.

[0059] Optionally, the target instruction is a stop loading instruction. During the loading task execution on the database side, the loading task is stopped in response to the target instruction from the loading tool.

[0060] Optionally, a preset threshold `exit_limit` can be set in the loading tool to indicate when the target statistical count reaches the preset threshold and exit the entire loading process. Setting the preset threshold provides an exit mechanism, typically indicating a high number of abnormal events. In this case, the loading tool wants to stop the loading task to investigate the cause of the problem and prevent subsequent invalid operations from wasting time and resources. A high number of abnormal events may indicate that the data to be loaded in the target table is correct, but the loading tool's coding method has a problem, causing the database to consider the data incorrect.

[0061] Optionally, the preset quantity threshold can be determined by the amount of data to be loaded. The larger the amount of data to be loaded, the larger the preset quantity threshold.

[0062] For example, if there are 100,000 rows of data to be loaded in the target table, and the encoding method is incorrect, then all the data to be loaded will be corrupted. Instead of loading all 100,000 rows and then checking the cause of the error, exiting the loading process in advance to check whether it is an operational error can greatly reduce time costs and resource consumption.

[0063] Optionally, if the target number of statistics does not meet the preset number of abnormal events, the database will continue to execute the loading task. After the task is completed, the cause of the abnormality will be queried in the specified storage area based on the abnormal information stored locally in the database or the abnormal attribute fields received by the loading tool.

[0064] Optionally, since the exception feedback method can be automatically adjusted according to the amount of remaining data to be loaded in the database loading task, when there is a lot of data to be loaded, i.e., in the early stage of the loading task, the database performs coarse-grained event statistics on a per-storage-area basis, and the loading tool directly merges the event statistics from multiple storage areas, improving the efficiency of the loading tool in determining the target statistical quantity. When there is a little data to be loaded, i.e., in the later stage of the loading task, the database does not record exception information, but provides real-time feedback of exception attribute fields to the loading tool. The loading tool records exception information and performs fine-grained quantity statistics on a per-exception-event basis, improving the granularity of the number of triggered target instructions and reducing the waste of computing resources. Meanwhile, the database's approach of continuing loading without handling exceptions or rolling back when abnormal events are triggered improves the loading efficiency of normal database data, avoids the database table bloat problem caused by rollback, and thus does not affect the performance of subsequent table operations. The flexible adjustment of the two exception feedback methods reduces the complexity of filtering abnormal data and improves the success rate of data loading and operational efficiency. In addition, since the target statistical number generated based on data anomalies generally does not meet the preset exception event quantity condition, it is insufficient to cause the loading tool to issue the target instruction to stop loading, and the database will continue to load. However, a large number of exceptions caused by loading tool problems such as coding problems will cause the loading tool and the database to quickly exit the loading process, thus optimizing the exception handling process.

[0065] Optionally, the exception feedback method is determined based on the amount of remaining unloaded data in the loading task, including: if the amount of remaining unloaded data in the loading task is greater than a preset data amount threshold, determining the first parameter item as the first feedback value; performing exception event statistics on the detected exception events based on the first feedback value to obtain the current event statistics count; and, if the sub-loading task for the specified storage area is completed, reporting the currently obtained event statistics count to the loading tool.

[0066] Optionally, the preset data volume threshold can be 30% of the total data volume to be loaded.

[0067] Optionally, before execution on the database side, the loading tool inputs loading options into the database through a connection established with the database. These options include a first parameter, a second parameter, and a third parameter. The value of the first parameter is automatically adjusted based on changes in the amount of remaining data to be loaded. For example, if the amount of remaining data exceeds a preset data threshold, the first parameter is set to a first feedback value. The value of the second parameter matches the value of the first parameter.

[0068] Optionally, the first parameter represents the abnormal feedback method, and the second parameter represents the information fed back to the loading tool that matches the abnormal feedback method.

[0069] For example, if the first parameter, log_error, is set to the first feedback value "on", it means that the database records the exception information of the abnormal event locally and provides feedback according to coarse granularity. Then, the second parameter, log_level, is set to "default", which means that the number of events currently obtained is reported to the loading tool.

[0070] Optionally, the third parameter indicates whether to continue loading when the database encounters an abnormal event. A value of "stop" indicates that loading will be stopped, and a value of "ignore" indicates that the exception will be ignored and loading will continue. In this invention, the default value of the third parameter is set to "ignore".

[0071] Optionally, based on the first feedback value, the database first stores the exception information in a local buffer, and at the same time, the number of exception events in the local buffer is incremented by 1 to obtain the current event statistics.

[0072] Optionally, after the sub-loading task for the specified storage area corresponding to this abnormal event has been completed, the event statistics for the entire specified storage area stored locally in the database can be sent directly to the loading tool.

[0073] Optionally, if the remaining amount of data to be loaded exceeds a preset data volume threshold, the first parameter is set to the first feedback value. This allows the database to locally record exception information and count the current number of events. The database does not need to report exception information to the loading tool in real time; it directly stores and counts the data, reducing the amount of data exchanged over the network and lessening the exception recording and parsing burden on the loading tool. Since the loading tool cannot obtain the number of exception events in real time, it must wait until the sub-loading of the specified storage area is complete before summarizing the counts across multiple storage areas.

[0074] Optionally, the exception feedback method is determined based on the amount of data remaining to be loaded in the loading task. This also includes: recording the exception information of the exception event in the database cache according to the first feedback value; and, when the sub-loading task for the specified storage area is completed, performing item statistics on multiple exception information corresponding to the specified storage area to obtain the current event statistics count, and feeding back the currently obtained event statistics count to the loading tool.

[0075] Optionally, based on the first feedback value, the exception information of the exception event is recorded in the local cache of the database. When the sub-loading task for the specified storage area is completed, the database performs an entry count on the multiple exception information in the specified cache area to obtain the current event count and reports the current event count to the loading tool.

[0076] Optionally, the method of determining the exception feedback based on the amount of data remaining to be loaded in the loading task may further include: if the amount of data remaining to be loaded in the loading task is less than or equal to a preset data amount threshold, determining the parameter value of the first parameter item as the second feedback value; and feeding back the exception attribute field of the exception event to the loading tool based on the second feedback value.

[0077] Optionally, if the amount of remaining data to be loaded is less than or equal to a preset data volume threshold, it means that most of the data to be loaded has been loaded. If the amount of remaining data to be loaded is less than or equal to the preset data volume threshold, the parameter value of the first parameter item is automatically adjusted to the second feedback value.

[0078] For example, if the first parameter item `log_error` is set to the second feedback value `off`, it means that the database does not record exception information of abnormal events locally and provides feedback in real time with fine granularity. Then, the second parameter item `log_level` is set to `verbose`, which means that the exception attribute fields are fed back to the loading tool.

[0079] Optionally, the exception attribute field may include information such as the line number of the exception event in the specified storage area, the exception field name, and the exception value.

[0080] For example, the feedback format to the loading tool is: NOTICE: skipping row at line 2 for column "n": "a". Here, line 2 indicates the second line in the specified storage area where the exception event occurs, n is the exception field name, and a is the exception value.

[0081] Optionally, as the amount of remaining data to be loaded decreases, there are already event statistics for multiple storage areas. With the addition of a few abnormal events, the preset abnormal event count condition may be met. Therefore, counting the number of abnormal events on a storage area basis will result in some resource waste and redundancy. It is necessary to count the number of abnormal events in real time on a single abnormal event basis to ensure accurate and fine-grained statistics. Then, when the event count just meets the preset abnormal event count condition, loading should be stopped immediately to minimize resource waste.

[0082] Optionally, the target statistical quantity can be determined based on at least one of the exception attribute field and the event statistics quantity, including: summing the event statistics quantities corresponding to multiple specified storage areas to obtain the area statistics quantity; and determining the area statistics quantity as the target statistical quantity.

[0083] Optionally, if the first parameter item log_error is set to the first feedback value on, the loading tool receives the event statistics for each of the multiple specified storage areas.

[0084] The statistics plugin on the loading tool accumulates the event statistics for each of the multiple specified storage areas to obtain the area statistics. The area statistics represent the cumulative number of abnormal events in the current multiple specified storage areas. The area statistics are then set as the target statistics.

[0085] Optionally, the target statistical quantity is determined based on at least one of the abnormal attribute field and the event statistical quantity, and further includes: in response to the abnormal attribute field of the received abnormal event, performing an auto-incrementing statistical count based on the statistical plugin on the loading tool to determine the single abnormal statistical quantity; and determining the target statistical quantity based on the sum of the district statistical quantity and the single abnormal statistical quantity.

[0086] Optionally, if the parameter value of the first parameter item is the historical period of the first feedback, the loading tool has already obtained the number of district statistics.

[0087] Optionally, based on the amount of remaining data to be loaded, the value of the first parameter item log_error can be adjusted to the second feedback value off, and the exception attribute field of the exception event can be uploaded to the loading tool in real time.

[0088] Optionally, in response to the exception attribute fields of received exception events, the loading tool performs an auto-incrementing count based on the statistics plugin on the loading tool to determine the single exception count. The single exception count is the cumulative number of single exception events received in real time.

[0089] Optionally, the district statistical count and the single anomaly statistical count can be summed to determine the target statistical count.

[0090] Figure 3 A flowchart illustrating the database-side loading task execution according to an embodiment of the present invention is shown.

[0091] like Figure 3 As shown, the database side performs loading tasks including operations S301 to S307.

[0092] In operation S301, the data to be loaded is processed one by one.

[0093] In operation S302, respond to the abnormal event and determine whether the value of the first parameter item is the first feedback value. If yes, proceed to operation S303; otherwise, proceed to operation S304.

[0094] Operation S303: The database records the exception information locally.

[0095] Operation S304: The database reports abnormal attribute fields in real time.

[0096] Operation S305: Accumulate the number of events.

[0097] In step S306, determine whether the processing is complete. If yes, proceed to step S302; otherwise, proceed to step S307.

[0098] Operation S307, end.

[0099] Optionally, the exception attribute fields include an exception attribute name field and a row number field of the specified storage area where the exception event is located. The database loading exception handling method also includes: if the first parameter value is the first feedback value, determining the original data from the data to be loaded based on the exception information recorded in the cache in the database; or if the first parameter value is the second feedback value, calling the loading tool to determine the original data from the specified storage area based on the row number field and the exception attribute name field; and analyzing the original data to obtain a processing strategy.

[0100] Optionally, the raw data can be extracted after the loading task has finished executing.

[0101] Optionally, the exception attribute name field can be a column attribute name, such as name, gender, etc.

[0102] Optionally, the exception information in the local cache of the database is exception data. If the first parameter value is the first feedback value, the exception location is determined directly from the original data to be loaded received from the storage area based on the exception information recorded in the cache of the database, and the original data at the exception location is extracted. The original data is the data in the data to be loaded; for example, if the exception data is 123~, the original data is 1234.

[0103] Optionally, if the first parameter value is the second feedback value, the loading tool parses the exception attribute name field and the row number field of the specified storage area where the exception event is located in the exception attribute field. It then determines the row of the storage area based on the row number field of the specified storage area where the exception event is located, and then determines the column of the storage area based on the exception attribute name field, thereby extracting the original data based on the row and column.

[0104] Optionally, error analysis will be performed on the original data to obtain processing strategies, such as modifying the abnormal information in the original data and reloading each item individually.

[0105] Optionally, loading abnormal data records and normal data during the loading process does not incur additional network or Central Processing Unit (CPU) consumption, meaning no additional retries are needed and therefore no extra overhead is involved. Since there is no process for filtering abnormal data, there will be no table bloat issues on the database side, saving physical storage in the cluster, and subsequent table operations will not suffer from inefficiency problems.

[0106] Figure 4 A flowchart illustrating the execution of a loading tool according to an embodiment of the present invention is shown.

[0107] like Figure 4 As shown, the loading tool execution process includes operations S401 to S404.

[0108] In operation S401, parse the abnormal attribute fields uploaded to the database.

[0109] In operation S402, determine the exception attribute name field and the line number field.

[0110] In operation S403, the raw data is determined from the specified storage area.

[0111] In operation S404, save the original data.

[0112] The optimization effects were analyzed from two dimensions: network and CPU. Network refers to the number of interactions between the loading tool and the database, while CPU refers to the amount of data the database processed.

[0113] Suppose there are M records to be loaded, of which N are abnormal records. In reality, M is much larger than N. Previous techniques, such as a record-by-record splitting algorithm, require retransmitting each of the M records when an anomaly occurs. This increases the number of network connections by M, and the amount of additional data the database needs to process is also M. This algorithm is only related to M, not N. Its advantage is its simplicity and ease of implementation, but its disadvantage is that network interaction is directly proportional to the amount of data. In practice, large data volumes can lead to significant network pressure.

[0114] Another related technique before optimization, such as using a binary splitting algorithm, takes N much smaller than M in practice as an example, requiring the following number of network additions: The amount of data to be processed varies depending on the location of the abnormal records, with a maximum of 2 + 2² + ... + = -2, the minimum is 1 + 2 + 2² + ... + + = + M -1. The binary search algorithm loads data from the storage area in two parts. This reduces network interaction for large datasets but increases database CPU consumption. Both algorithms rely on continuous attempts to filter out abnormal data, increasing network and CPU usage and consuming cluster resources. During the anomaly filtering process, continuously deleting inserted table records and re-uploading records means that metadata records in the table's logical storage cannot be deleted in a timely manner, leading to an increase in invalid data, table bloat, wasted storage resources, and significantly reduced efficiency of subsequent table operations.

[0115] For example, if there are 1024 data entries in the storage area, one of which is an abnormal record, the uni-row splitting algorithm would require 1024 new network interactions and 1024 CPU operations; the binary splitting algorithm would require 20 new network interactions and a minimum of 1033 to a maximum of 2046 CPU operations. During both processes, the database tables would have already had a significant amount of invalid data inserted, leading to table bloat. This invention, however, can simultaneously handle exception feedback and normal data loading during the loading process. When an exception event is triggered, information such as the row number and name of the abnormal data is returned to the loading tool, and the loading task continues. This avoids the problem of reloading data after filtering exceptions. Therefore, because the loading task is uninterrupted, it does not involve additional network or CPU operations, and there is no table bloat problem.

[0116] Table 1 shows a comparison of the effects according to an embodiment of the present invention.

[0117]

[0118] Based on the above-described database loading exception handling method, this invention also provides a database loading exception handling device. The following will be combined with... Figure 5 The device is described in detail.

[0119] Figure 5 A structural block diagram of a database loading exception handling apparatus according to an embodiment of the present invention is shown.

[0120] like Figure 5 As shown, the database loading exception handling device 500 of this embodiment includes an execution module 510, a response module 520, and a stop module 530.

[0121] The execution module 510 is used to perform a loading task for the data to be loaded. In one embodiment, the execution module 510 can be used to perform the operation S210 described above, which will not be repeated here.

[0122] The response module 520 is used to respond to an exception event triggered during the execution of a loading task. It determines the exception feedback method based on the amount of remaining data to be loaded in the loading task. The exception feedback method may include sending the exception attribute field of the exception event to the loading tool, or counting the number of exception events and, upon completion of the sub-loading task for the specified storage area, reporting the current number of events to the loading tool. In one embodiment, the response module 520 may be used to perform the operation S220 described above, which will not be repeated here.

[0123] The stop module 530 is used to stop the loading task in response to a target instruction from the loading tool during the execution of the loading task. The loading tool determines the target statistical quantity based on at least one of the exception attribute field and the event statistics quantity, and sends a target instruction to the database if the target statistical quantity meets a preset exception event quantity condition. In one embodiment, the stop module 530 can be used to execute the operation S230 described above, which will not be repeated here.

[0124] Optionally, the response module 520 includes a first response submodule, a second response submodule, and a third response submodule.

[0125] The first response submodule is used to determine the first parameter value as the first feedback value when the amount of remaining data to be loaded in the loading task is greater than a preset data amount threshold.

[0126] The second response submodule is used to perform abnormal event statistics on the detected abnormal events based on the first feedback value, and obtain the current event statistics count.

[0127] The third response submodule is used to report the number of events currently acquired to the loading tool after the sub-loading task for the specified storage area has been completed.

[0128] Optionally, the response module 520 may also include a fourth response submodule and a fifth response submodule.

[0129] The fourth response submodule is used to record the exception information of the exception event in the database cache based on the first feedback value.

[0130] The fifth response submodule is used to count the number of exceptions in the specified storage area after the sub-loading task for the specified storage area has been completed, obtain the current event count, and report the current event count to the loading tool.

[0131] Optionally, response module 520 may also include a sixth response submodule and a seventh response submodule.

[0132] The sixth response submodule is used to determine the value of the first parameter item as the second feedback value when the amount of data to be loaded in the loading task is less than or equal to a preset data amount threshold.

[0133] The seventh response submodule is used to provide feedback on the exception attribute fields of the exception event to the loading tool based on the second feedback value.

[0134] Optionally, the response module 520 may also include a first accumulation submodule and a first statistics submodule.

[0135] The first accumulation submodule is used to accumulate the event statistics corresponding to multiple specified storage areas to obtain the area statistics.

[0136] The first statistics submodule is used to determine the district statistics quantity as the target statistics quantity.

[0137] Optionally, the response module 520 may also include a second accumulation submodule and a second statistics submodule.

[0138] The second accumulation submodule is used to respond to the exception attribute field of the received exception event, and perform auto-incrementing statistics based on the statistics plugin on the loading tool to determine the number of single exception statistics.

[0139] The second statistics submodule is used to determine the target statistics quantity based on the sum of the district statistics quantity and the single anomaly statistics quantity.

[0140] Optionally, the database loading exception handling 500 may also include a first determination module, a second determination module, and an analysis module.

[0141] The first determination module is used to determine the original data from the data to be loaded based on the exception information recorded by the cache in the database when the parameter value of the first parameter item is the first feedback value.

[0142] The second determination module is used to call the loading tool when the value of the first parameter item is the second feedback value, and determine the original data from the specified storage area based on the row number field and the exception attribute name field.

[0143] The analysis module is used to analyze the raw data and derive processing strategies.

[0144] Optionally, any plurality of modules among execution module 510, response module 520, and stop module 530 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules may be combined with at least a portion of the functionality of other modules and implemented in one module. Optionally, at least one of execution module 510, response module 520, and stop module 530 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of execution module 510, response module 520, and stop module 530 may be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0145] Figure 6 A block diagram of an electronic device suitable for implementing a database loading exception handling method according to an embodiment of the present invention is shown.

[0146] Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0147] like Figure 6 As shown, a computer electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a ROM 602 (read-only memory) or a program loaded from a storage portion 608 into a RAM 603 (random access memory). The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0148] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 602 and / or RAM 603. It should be noted that programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in one or more memories.

[0149] Optionally, the electronic device 600 may also include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0150] Optionally, the method flow according to embodiments of the present invention can be implemented as a computer software program. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by processor 601, it performs the functions defined in the system of embodiments of the present invention. Optionally, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0151] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the database loading exception handling method according to embodiments of the present invention.

[0152] Optionally, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0153] For example, optionally, the computer-readable storage medium may include ROM 602 and / or RAM 603 and / or one or more memories other than ROM 602 and RAM 603 as described above.

[0154] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of the present invention. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the database loading exception handling method provided in the embodiments of the present invention.

[0155] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this embodiment of the invention. Optionally, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0156] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0157] Optionally, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0158] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or pairings fall within the scope of this invention.

[0159] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A method for handling database loading exceptions, characterized in that, Applied to the database side, the method includes: Execute the loading task for the data to be loaded; In response to an exception event triggered during the execution of a loading task, the exception feedback method is determined based on the amount of remaining unloaded data in the loading task, including: If the amount of remaining data to be loaded in the loading task is greater than a preset data amount threshold, the parameter value of the first parameter item is determined to be the first feedback value. Based on the first feedback value, perform anomaly statistics on the detected abnormal events to obtain the current event count; Once the sub-loading task targeting the specified storage area has been completed, the loading tool will be informed of the number of events currently acquired. If the amount of remaining data to be loaded in the loading task is less than or equal to a preset data amount threshold, the parameter value of the first parameter item is determined to be the second feedback value. The abnormal attribute field of the abnormal event is fed back to the loading tool according to the second feedback value. The abnormal feedback method means sending the abnormal attribute field of the abnormal event to the loading tool, or counting the event statistics of the abnormal event, and feeding back the current event statistics to the loading tool after completing the sub-loading task for the specified storage area. During the execution of the loading task, in response to the target instruction from the loading tool, the loading task is stopped, wherein the loading tool determines the target statistical quantity based on at least one of the abnormal attribute field and the event statistical quantity, and sends the target instruction to the database if the target statistical quantity meets the preset abnormal event quantity condition.

2. The method according to claim 1, characterized in that, The method for determining the exception feedback based on the amount of remaining unloaded data in the loading task also includes: Based on the first feedback value, record the abnormal information of the abnormal event in the database cache; When the sub-loading task for the specified storage area is completed, the multiple exception information items corresponding to the specified storage area are counted to obtain the current event count, and the current event count is fed back to the loading tool.

3. The method according to claim 1, characterized in that, The target statistical quantity is determined based on at least one of the abnormal attribute field and the event statistical quantity, including: The event statistics corresponding to each of the multiple specified storage areas are summed to obtain the area statistics; The statistical quantity of the aforementioned district is determined as the target statistical quantity.

4. The method according to claim 3, characterized in that, Determining the target statistical quantity based on at least one of the abnormal attribute field and the event statistical quantity further includes: In response to the received abnormal event's abnormal attribute field, the number of single abnormal events is determined by performing an auto-incrementing count based on the statistics plugin on the loading tool. The target statistical quantity is determined based on the sum of the district statistical quantity and the single anomaly statistical quantity.

5. The method according to claim 1, characterized in that, The exception attribute field includes an exception attribute name field and a row number field of the specified storage area where the exception event is located. The method further includes: If the first parameter value is the first feedback value, the original data is determined from the data to be loaded based on the exception information recorded in the cache in the database; or If the value of the first parameter item is the second feedback value, the loading tool is invoked to determine the original data from the specified storage area based on the row number field and the abnormal attribute name field. The raw data is analyzed to obtain a processing strategy.

6. A database loading exception handling device, characterized in that, The device, applied to the database side, includes: The execution module is used to perform the loading task for the data to be loaded. The response module is used to respond to an abnormal event triggered by the execution of a loading task. It determines the abnormal feedback method based on the amount of remaining data to be loaded in the loading task. The abnormal feedback method may include sending the abnormal attribute field of the abnormal event to the loading tool, or counting the event statistics of the abnormal event and, upon completion of the sub-loading task for a specified storage area, reporting the current event statistics to the loading tool. The response module includes a first response submodule, a second response submodule, a third response submodule, a sixth response submodule, and a seventh response submodule. The first response submodule is used to determine the first parameter value as the first feedback value when the amount of remaining data to be loaded in the loading task is greater than a preset data amount threshold. The second response submodule is used to perform abnormal event statistics on the detected abnormal events based on the first feedback value, and obtain the current event statistics count. The third response submodule is used to report the number of events currently acquired to the loading tool after the sub-loading task for the specified storage area has been completed. The sixth response submodule is used to determine the value of the first parameter item as the second feedback value when the amount of data remaining to be loaded in the loading task is less than or equal to a preset data amount threshold. The seventh response submodule is used to provide feedback on the abnormal attribute field of the abnormal event to the loading tool based on the second feedback value; The stop module is used to stop the loading task in response to a target instruction from the loading tool during the execution of the loading task, wherein the loading tool determines the target statistical quantity based on at least one of the abnormal attribute field and the event statistical quantity, and sends the target instruction to the database when the target statistical quantity meets a preset abnormal event quantity condition.

7. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors invoke the one or more computer programs to implement the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is invoked by the processor, it implements the steps of the method according to any one of claims 1 to 5.

Images