Data processing method and system, electronic device and storage medium

Abstract

This application discloses a data processing method, system, electronic device, and storage medium. The data processing method includes: acquiring a data task, wherein the data task represents a task of synchronizing target data from a source database to a target database; and executing the data task to synchronize the target data from the source database to the target database; wherein the data volume of the target data table in the target database is smaller than the data volume of the source data table in the source database. This solution achieves the acquisition of a small range of data by single task, eliminating the need to read the entire dataset, enabling rapid adaptation, improving data query speed, and reducing storage space. Simultaneously, by executing the data task to synchronize the target data from the source database to the target database, flexible database switching is achieved, improving database adaptability.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, system, electronic device, and storage medium. Background Technology

[0002] In the current battery industry, there are usually high requirements for data display during the delivery process. This data typically includes raw battery data, battery characteristic data, fault data, status data, etc.

[0003] For example, in a battery CT (Computed Tomography) scenario, battery dashboards typically need to have visualization capabilities for this data. However, due to differences in data types, data structures, and storage media, there are usually significant differences in the display methods. Therefore, building visualization capabilities requires substantial manpower to quickly develop or adapt to various types of data, making the ability to quickly develop or adapt to various types of data crucial. Summary of the Invention

[0004] This application provides at least one data processing method, system, electronic device, and storage medium.

[0005] This application provides a data processing method, comprising: acquiring a data task, wherein the data task represents a task of synchronizing target data from a source database to a target database; executing the data task to synchronize the target data from the source database to the target database; wherein the data volume of the target data table in the target database is smaller than the data volume of the source data table in the source database.

[0006] In the above solution, data acquisition tasks are performed to synchronize target data from the source database to the target database. The target database contains fewer data tables than the source database contains tables, allowing for the acquisition of small datasets as individual tasks, eliminating the need to read the entire dataset, enabling rapid adaptation, improved data query speed, and reduced storage space. Furthermore, by executing data tasks to synchronize target data from the source database to the target database, flexible database switching is achieved, enhancing database adaptability.

[0007] In some embodiments, executing the data task includes: encoding the task information corresponding to the data task to obtain encoded task information; obtaining the code identifier associated with the encoded task information; and processing the encoded task information and the code identifier to synchronize the target data from the source database to the target database.

[0008] In the above solution, the task information corresponding to the data task is encoded to obtain the encoded task information, and the code identifier associated with the encoded task information is obtained. Then, the data is processed according to the encoded task information and the code identifier to synchronize the target data from the source database to the target database. The database type can be switched at any time, which solves the problem of database type limitation in the battery CT scenario, thereby realizing flexible database switching and improving database adaptability.

[0009] In some embodiments, the processing based on the encoding task information and the code identifier includes: obtaining a preset task corresponding to the code identifier based on the code identifier; and executing the preset task to synchronize the target data from the source database to the target database based on the encoding task information.

[0010] In the above solution, by obtaining the preset task corresponding to the code identifier based on the code identifier, and then executing the preset task, the target data is synchronized from the source database to the target database according to the encoded task information. The database type can be switched at any time, solving the problem of database type limitation in the battery CT scenario, thereby realizing flexible database switching and improving database adaptability.

[0011] In some embodiments, synchronizing the target data from the source database to the target database based on the encoding task information includes: parsing the encoding task information to obtain task information, wherein the task information includes at least connection information corresponding to the source database, field information of the target data, data aggregation information, and connection information corresponding to the target database; establishing a connection with the source database according to the connection information corresponding to the source database; obtaining raw data from the source database according to the field information of the target data; performing an aggregation operation on the raw data according to the data aggregation information to obtain the target data; and writing the target data into a target data table in the target database according to the connection information corresponding to the target database.

[0012] In the above solution, by parsing the encoding task information, the task information can be obtained, enabling the switching of database types at any time. This solves the limitation of database types in battery CT scenarios, thereby achieving flexible database switching and improving database adaptability. Simultaneously, by writing target data into the target data table of the target database based on the task information, automated data processing is achieved, maintaining efficiency and reliability while improving data security and accuracy.

[0013] In some embodiments, the data processing method further includes: displaying the target data on a preset interface.

[0014] In the above solution, data visualization is achieved by displaying the target data on a preset interface, thus meeting the data visualization requirements.

[0015] In some embodiments, the target database includes at least one target data table, wherein the at least one target data table corresponds to different execution cycles of the data task.

[0016] In the above scheme, different data requirements are met by using the different execution cycles of data tasks corresponding to at least one target data table in the target database.

[0017] In some embodiments, displaying the target data on a preset interface includes: switching between querying the at least one target data table to obtain the target data; and switching between displaying the at least one target data table on the preset interface to display the target data.

[0018] In the above solution, by switching to query at least one target data table and displaying at least one target data table on a preset interface, the target data is displayed. Since the amount of data in each target data table is less than the amount of data in the source data table of the source database, the computational burden of filtering redundant data and data aggregation operations is effectively reduced, thereby improving the speed of data query and the stability of the interface in the battery CT scenario, reducing the query load of the database, and improving system performance.

[0019] In some embodiments, the data acquisition task includes: configuring and / or updating task information corresponding to the data task to obtain the data task; and / or scanning a fault list to obtain the data task.

[0020] In the above solution, data tasks are obtained by configuring and / or updating the task information corresponding to the data tasks, and / or scanning the fault list. This allows for the acquisition of small-scale data on a single task basis, eliminating the need to read the entire dataset, enabling rapid adaptation, improving data query speed, and reducing storage space. Furthermore, by configuring and / or updating task information and / or scanning the fault list, customized data table structures can be dynamically generated according to the user's specific needs. This allows different users to generate different data table structures, thereby meeting personalized data display and analysis requirements.

[0021] In some embodiments, the task information includes at least the connection information corresponding to the source database, the connection information corresponding to the target database, the field information of the target data, the data aggregation information corresponding to the target data, and the device information corresponding to the target data; configuring the task information corresponding to the data task includes: configuring the connection information corresponding to the source database and the connection information corresponding to the target database through a first interface; configuring the field information of the target data, the data aggregation information, and the device information corresponding to the target data through a second interface, thereby obtaining the data task.

[0022] In the above solution, the connection information of the source database and the target database are configured through the first interface, and the field information, data aggregation information and device information of the target data are configured through the second interface, thereby obtaining the data task and realizing the visualization configuration of the data task. It can dynamically generate customized data table structures according to the user's specific needs, so that different users can generate different data table structures, thereby meeting personalized data display and analysis needs.

[0023] In some embodiments, the task information includes the display method of the target data; configuring the task information corresponding to the data task includes configuring the display method of the target data.

[0024] In the above solution, by configuring the display method of target data, the visualization configuration data task can be further realized. It can dynamically generate customized data table structures according to the specific needs of users, so as to generate different data table structures for different users, thereby meeting personalized data display and analysis needs.

[0025] In some embodiments, the step of scanning the fault list to obtain the data task includes: scanning the fault list according to a preset period to obtain the data task.

[0026] In the above solution, by periodically scanning the fault list to obtain data tasks, customized data table structures can be dynamically generated according to the user's specific needs, so as to generate different data table structures for different users, thereby meeting personalized data display and analysis needs.

[0027] In some embodiments, the data processing method further includes: recording the data tasks and their task status in a preset task list.

[0028] In the above scheme, data tasks and their status are recorded in a preset task list, which enables efficient and flexible management of data tasks while maintaining effective control over the corresponding data.

[0029] In some embodiments, in response to acquiring the data task, the task state of the data task is a waiting state; in response to executing the data task, the task state of the data task is a start state; in response to canceling the data task, the task state of the data task is a canceled state.

[0030] The above solution achieves efficient and flexible management of data tasks by setting different task states for data tasks, while maintaining effective control over the corresponding data.

[0031] In some embodiments, recording the data task and its task status in a preset task list includes: in response to executing the data task and successfully synchronizing the target data from the source database to the target database, recording the task status of the data task as a success status in the preset task list; in response to executing the data task and failing to successfully synchronize the target data from the source database to the target database within a preset time, recording the task status of the data task as a failure status in the preset task list.

[0032] In the above scheme, when a data task is executed and the target data is successfully synchronized from the source database to the target database, the task status of the data task is recorded as a success in the preset task list. When a data task is executed but the target data is not successfully synchronized from the source database to the target database within a preset time, the task status of the data task is recorded as a failure in the preset task list. This achieves effective task monitoring, maintains the accuracy and timeliness of task status, and thus enables efficient and flexible management of data tasks while maintaining effective control over the corresponding data.

[0033] In some embodiments, recording the data task and its task status in a preset task list further includes: updating the task status of the data task in the preset task list from a successful state to an expired state in response to a first preset time when the data task is in a successful state; updating the task status of the data task in the preset task list from the failed state to an expired state in response to a second preset time when the data task is in a failed state; wherein the second preset time is less than the first preset time.

[0034] In the above scheme, the task status of the data task is updated from successful to expired after a first preset time when the data task is in a successful state, and the task status of the data task is updated from failed to expired after a second preset time when the data task is in a failed state. This further enables effective task monitoring, maintains the accuracy and timeliness of task status, and thus achieves efficient and flexible management of data tasks while maintaining effective control over the corresponding data.

[0035] In some embodiments, the data processing method further includes: in response to the data task being in an expired state for a third preset time, deleting the target data table and / or the corresponding target data corresponding to the data task.

[0036] In the above scheme, by responding to the third preset time when the data task is in an expired state, the target data table and / or the corresponding target data corresponding to the data task are deleted, so as to keep the target database clean and efficient, effectively utilize storage space, and realize timely data updates and effective management.

[0037] This application provides a data processing system, including: a task acquisition module for acquiring data tasks, wherein the data task represents a task of synchronizing target data from a source database to a target database; and a task execution module for executing the data task to synchronize the target data from the source database to the target database; wherein the data volume of the target data table in the target database is smaller than the data volume of the source data table in the source database.

[0038] This application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-described data processing method.

[0039] This application provides a computer-readable storage medium storing program instructions thereon, which, when executed by a processor, implement the above-described data processing method.

[0040] In the above solution, data acquisition tasks are performed to synchronize target data from the source database to the target database. The target database contains fewer data tables than the source database contains tables, allowing for the acquisition of small datasets as individual tasks, eliminating the need to read the entire dataset. This enables rapid adaptation, improves data query speed, and reduces storage space. Furthermore, synchronizing target data from the source database to the target database through data acquisition tasks allows for flexible database switching and enhances database adaptability.

[0041] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0043] Figure 1 This is a schematic flowchart of a data processing method provided in some embodiments of this application;

[0044] Figure 2This is a partial flowchart illustrating another data processing method provided in some embodiments of this application;

[0045] Figure 3 This is a schematic diagram illustrating the format of the encoding task information for data tasks provided in some embodiments of this application;

[0046] Figure 4 This is a partial flowchart illustrating another data processing method provided in some embodiments of this application;

[0047] Figure 5 This is a schematic flowchart illustrating the data provided in some embodiments of this application;

[0048] Figure 6 These are schematic diagrams of the interface for configuring data tasks provided in some embodiments of this application;

[0049] Figure 7 This is a schematic diagram of the data processing flow provided in some embodiments of this application;

[0050] Figure 8 This is a schematic diagram of a data task configuration page provided in some embodiments of this application;

[0051] Figure 9 This is a schematic diagram illustrating the lifecycle of a data task provided in some embodiments of this application;

[0052] Figure 10 This is a schematic diagram illustrating information transmission during the execution of data tasks provided in some embodiments of this application;

[0053] Figure 11 This is a schematic diagram of the structure of a data processing system provided in some embodiments of this application;

[0054] Figure 12 These are schematic diagrams of the structure of electronic devices provided in some embodiments of this application;

[0055] Figure 13 This is a schematic diagram of the structure of a computer-readable storage medium provided in some embodiments of this application. Detailed Implementation

[0056] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0057] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0058] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0059] In the current battery industry, there are typically high requirements for data visualization during the delivery process. This data usually includes raw battery data, battery characteristic data, fault data, and status data. In battery CT scenarios, it is generally necessary to have the ability to visualize raw battery data, battery characteristic data, fault data, and status data. However, due to differences in data types, data structures, and storage media, there are usually significant differences in the visualization methods. Therefore, building visualization capabilities requires substantial manpower to quickly develop or adapt to various types of data, making the ability to quickly develop or adapt to various types of data crucial.

[0060] To address this, this application employs a data acquisition task, which is then executed to synchronize target data from the source database to the target database. The target data table in the target database contains less data than the source data table in the source database. This allows for the acquisition of a small range of data per task, eliminating the need to read the entire dataset, enabling rapid adaptation, improved data query speed, and reduced storage space. Furthermore, by executing the data task to synchronize target data from the source database to the target database, flexible database switching is achieved, enhancing database adaptability.

[0061] Please see Figure 1 , Figure 1 This is a flowchart illustrating a data processing method provided in some embodiments of this application. The data processing method can be applied to battery devices, such as faulty battery devices in a battery CT scenario, where the faulty battery device can be a cell, battery, battery cabinet, etc. The executing entity of the data processing method can be an electronic device with computing capabilities, such as a microcomputer, server, and mobile devices such as laptops and tablets. Figure 1 As shown, the data processing method includes: Step S11: Obtaining a data task, where the data task represents a task to synchronize target data from the source database to the target database. Step S12: Executing the data task to synchronize the target data from the source database to the target database; wherein the data volume of the target data table in the target database is smaller than the data volume of the source data table in the source database.

[0062] A data task refers to the task of synchronizing target data from a source database to a target database. In other words, a data task can also be called a data synchronization task or a data retrieval task.

[0063] The source database can be stored in a storage engine. For example, the source data tables in the source database can be Hive tables stored in the Hive storage engine. Hive is a data warehouse tool used for data extraction, transformation, and loading; it is an engine that can store, query, and analyze large-scale data stored in Hadoop. In the battery CT scenario, the source database can include raw data from faulty battery devices, such as fault warning data, status data, algorithm results data, and feature data.

[0064] The target database can be a MySQL database, a Doris database, a ClickHouse database, etc. Different target data tables in the target database can correspond to different execution cycles of the data task. That is, the data task is executed once per cycle, and the corresponding target data is synchronized to the corresponding target data table. For example, target data table a corresponds to execution cycle b. Every time data task b is executed, the corresponding target data is synchronized to target table a.

[0065] The target data table in the target database contains less data than the source data table in the source database. In other words, the source data table in the source database contains more data than the target data table in the target database. Therefore, the target data obtained after data processing is less than the data in the source data table in the source database. Consequently, because the target data table in the target database contains less data than the source data table in the source database, the storage space for the target data table in the target database is less than the storage space for the source data table in the source database.

[0066] For example, a data task might be to query the recent maximum, minimum, and average values ​​of voltage, current, and sensor temperature for a faulty battery device, as shown in Table 1. The source data table for faulty battery device 1 contains 846k rows of data. After the data task, target data table 1 for faulty battery device 1 contains 19k rows, target data table 2 contains 4k rows, target data table 3 contains 1.8k rows, and target data table 4 contains 0.47k rows. It can be seen that the target database for faulty battery device 1 contains 25.27k rows, which is significantly less than the number of rows in its source data table (846k rows). Similarly, the source data table for faulty battery device 2 contains 7643k rows of data. After the data task, target data table 1 for faulty battery device 1 contains 14k rows, target data table 2 contains 4k rows, target data table 3 contains 1.79k rows, and target data table 4 contains 0.47k rows. It can be seen that the target database for faulty battery device 2 contains 20.26k rows, which is significantly less than the number of rows in its source data table (7643k rows). Among them, target data tables 1-4 correspond to different execution cycles of data tasks. For example, the execution cycle of target data table 1 is 10 minutes, the execution cycle of target data table 2 is 1 hour, the execution cycle of target data table 3 is 4 hours, and the execution cycle of target data table 4 is 1 day.

[0067] Table 1 Comparison of data before and after the faulty battery device.

[0068] Faulty battery device 1 Faulty battery device 2 Source data table 846k 7643k Target Data Table 1 19k 14k Target Data Table 2 4k 4k Target Data Table 3 1.8k 1.79k Target Data Table 4 0.47k 0.47k

[0069] In the above solution, data acquisition tasks are performed to synchronize target data from the source database to the target database. The target database contains fewer data tables than the source database contains tables, allowing for the acquisition of small datasets as individual tasks, eliminating the need to read the entire dataset, enabling rapid adaptation, improved data query speed, and reduced storage space. Furthermore, by executing data tasks to synchronize target data from the source database to the target database, flexible database switching is achieved, enhancing database adaptability.

[0070] In some embodiments, please refer to Figure 2 In step S12, the data task execution includes: Step S21: Encoding the task information corresponding to the data task to obtain encoded task information. Step S22: Obtaining the code identifier associated with the encoded task information. Step S23: Processing based on the encoded task information and the code identifier to synchronize the target data from the source database to the target database.

[0071] Data tasks can be executed by a processing end and a server. When the execution entity of the data processing method is an electronic device, the electronic device executes the data task. The processing end can be the processor of the electronic device, and the server can be the pipeline processor of the electronic device. During the execution of the data task, the processing end executes steps S21 and S22, transmitting the encoding task information and its associated code identifier to the server. The server receives the encoding task information and its associated code identifier and executes step S23, synchronizing the target data from the source database to the target database. Of course, the execution of the data task can also be achieved by two electronic devices working together. For example, one electronic device acts as the processing end, executing steps S21 and S22 to transmit the encoding task information and its associated code identifier to another electronic device. The other electronic device acts as the server, receiving the encoding task information and its associated code identifier and executing step S23, synchronizing the target data from the source database to the target database.

[0072] The task information for a data task can be configured during the data task setup process, for example, through a visual interface. This task information can include at least the connection information for the source database, the field information for the target data, the data aggregation information for the target data, and the connection information for the target database. The connection information for the source database can include the table name and table type. The field information for the target data can include the voltage, current, and sensor temperature of the faulty battery device, and the vehicle speed, vehicle status, and accumulated mileage of the vehicle corresponding to the faulty battery device. The data aggregation information for the target data can include the maximum, minimum, average, and median values; additionally, it can include values ​​obtained using predefined formulas. The connection information for the target database can include the table type, table name, and link information such as a URL for the target data table.

[0073] If the task information is encoded using base64 encoding, then the encoded task information can be in base64 format. For example, the encoded task information corresponding to a certain data task might be as follows: Figure 3 As shown.

[0074] In the above solution, the task information corresponding to the data task is encoded to obtain the encoded task information, and the code identifier associated with the encoded task information is obtained. Then, the data is processed according to the encoded task information and the code identifier to synchronize the target data from the source database to the target database. The database type can be switched at any time, which solves the problem of database type limitation in the battery CT scenario, thereby realizing flexible database switching and improving database adaptability.

[0075] In some embodiments, in step S23, processing based on encoding task information and code identifier includes: obtaining a preset task corresponding to the code identifier based on the code identifier; and executing the preset task to synchronize target data from the source database to the target database based on the encoding task information.

[0076] The default task is either a Spark Stream or a Spark Task. Based on the code identifier, the server invokes the corresponding Spark Stream or Spark Task to execute it, synchronizing the target data from the source database to the target database according to the encoded task information, such as base64 encoding.

[0077] In the above solution, by obtaining the preset task corresponding to the code identifier based on the code identifier, and then executing the preset task, the target data is synchronized from the source database to the target database according to the encoded task information. The database type can be switched at any time, solving the problem of database type limitation in the battery CT scenario, thereby realizing flexible database switching and improving database adaptability.

[0078] In some embodiments, please refer to Figure 4 Based on the encoding task information, the target data is synchronized from the source database to the target database, including: Step S41: Parsing the encoding task information to obtain task information, which includes at least the connection information corresponding to the source database, the field information of the target data, data aggregation information, and the connection information corresponding to the target database. Step S42: Establishing a connection with the source database according to the connection information corresponding to the source database. Step S43: Obtaining the original data from the source database based on the field information of the target data. Step S44: Performing an aggregation operation on the original data according to the data aggregation information to obtain the target data. Step S45: Writing the target data into the target data table in the target database according to the connection information corresponding to the target database.

[0079] Task information can include at least the connection information of the source database, the field information of the target data, the data aggregation information of the target data, and the connection information of the target database. The connection information of the source database can include table names, table types, etc. The field information of the target data can include the voltage, current, and sensor temperature of the faulty battery device, and the vehicle speed, vehicle status, and cumulative mileage of the vehicle corresponding to the faulty battery device. The data aggregation information of the target data can include maximum, minimum, average, and median values; it can also include values ​​obtained according to predefined formulas. The connection information of the target database can include the table type, table name, and link information of the target data table. The parsed encoded task information, i.e., the task information, can be in JSON format. For example, a portion of the JSON format of the parsed encoded task information for a certain data task might look like this:

[0080]

[0081]

[0082] In step S43, based on the field information of the target data, the original data is obtained from the source database. Specifically, an SQL query statement can be constructed based on the field information of the target data, and the constructed SQL query statement is used to extract the original data from the source database. The SQL query statement can be determined through predefined logic. For example, if the connection information of the source database is a Hive table, the field information of the target data is the voltage of the faulty battery device, the data aggregation information of the target data is the maximum value of the voltage of the faulty battery device, and the connection information of the target database is a data table in MySQL, then the voltage data of the faulty battery device is queried from the Hive table using an SQL query statement, and the aggregation operation corresponding to the "maximum value" is performed on these data to obtain the aggregated data, i.e., the target data, and then the aggregated data is written to the data table in MySQL.

[0083] In the above solution, by parsing the encoding task information, the task information can be obtained, enabling the switching of database types at any time. This solves the limitation of database types in battery CT scenarios, thereby achieving flexible database switching and improving database adaptability. Simultaneously, by writing target data into the target data table of the target database based on the task information, automated data processing is achieved, maintaining efficiency and reliability while improving data security and accuracy.

[0084] In some embodiments, the data processing method further includes: displaying the target data on a preset interface.

[0085] The default interface can be any interface on the processing end of the electronic device. The display method of the target data can be configured when configuring the data task. The target data can be displayed directly on the default interface. Alternatively, the target data can be processed first, such as by creating charts, and then displayed in the form of charts.

[0086] For example, a data task could be to query the maximum, minimum, and average speeds of the vehicle containing the faulty battery device. Figure 5 As shown, the target data, namely the maximum, minimum, and average vehicle speeds, are displayed using a line graph.

[0087] In the above solution, data visualization is achieved by displaying the target data on a preset interface, thus meeting the data visualization requirements.

[0088] In some embodiments, the target database includes at least one target data table, wherein the at least one target data table corresponds to different execution cycles of the data task.

[0089] Different target data tables in the target database can correspond to different execution cycles of the data task. That is, the data task is executed once per cycle, and the corresponding target data is synchronized to the corresponding target data table. For example, target data table 'a' corresponds to execution cycle 'b'. Every time data task 'b' is executed, the corresponding target data is synchronized to target data table 'a'. The execution cycle can be referred to as the time granularity; that is, different target data tables correspond to different time granularities. The different execution cycles of the data task are within the data time range of the faulty battery device, for example, within a preset time before and after the faulty battery device malfunctions. If the data volume of the target data tables in the target database is less than the data volume of the source data tables in the source database, then at least one target data table has a smaller data volume than the source database's source tables.

[0090] In the above scheme, different data requirements are met by using the different execution cycles of data tasks corresponding to at least one target data table in the target database.

[0091] In some embodiments, displaying target data on a preset interface includes: switching to query at least one target data table to obtain target data; and switching to display at least one target data table on the preset interface to display the target data.

[0092] Switching between querying at least one target data table can be done based on the display requirements of a preset interface. The data obtained from querying a specific target data table can be all the data in that table or a portion of it. Different target data tables correspond to different time granularities. For example, if the data task is to query the vehicle speed corresponding to a faulty battery device, after executing the data task according to different time granularities, the corresponding vehicle speed data will be synchronized to the target data tables with different time granularities. For instance, target data table a1 corresponds to a time granularity of 10 minutes, meaning that a data task is executed every 10 minutes, and the corresponding data is synchronized to target data table a1; target data table a2 corresponds to a time granularity of 1 hour, meaning that a data task is executed every hour, and the corresponding data is synchronized to target data table a2; target data table a3 corresponds to a time granularity of 4 hours, meaning that a data task is executed every 4 hours, and the corresponding data is synchronized to target data table a3, and so on. Accordingly, the storage space for target data table a1 is 11M, the storage space for target data table a2 is 2.8M, and the storage space for target data table a3 is 1.7M.

[0093] In the above solution, by switching to query at least one target data table and displaying at least one target data table on a preset interface, the target data is displayed. Since the amount of data in each target data table is less than the amount of data in the source data table of the source database, the computational burden of filtering redundant data and data aggregation operations is effectively reduced, thereby improving the speed of data query and the stability of the interface in the battery CT scenario, reducing the query load of the database, and improving system performance.

[0094] In some embodiments, obtaining a data task includes: configuring and / or updating task information corresponding to the data task to obtain the data task; and / or scanning a fault list to obtain the data task.

[0095] Configuring data task information can be done visually, for example, by configuring it on the interface of an electronic device. Once configured, the data task can be generated. Similarly, updating data task information can also be done visually, for example, by setting up an update control on the interface of an electronic device for updating. Clicking the update control will generate the data task.

[0096] The fault list is a list of faulty battery devices. Scanning the fault list can be done periodically, or the electronic device can automatically acquire data once the fault list is scanned.

[0097] In the above solution, data tasks are obtained by configuring and / or updating the task information corresponding to the data tasks, and / or scanning the fault list. This allows for the acquisition of small-scale data on a single task basis, eliminating the need to read the entire dataset, enabling rapid adaptation, improving data query speed, and reducing storage space. Furthermore, by configuring and / or updating task information and / or scanning the fault list, customized data table structures can be dynamically generated according to the user's specific needs. This allows different users to generate different data table structures, thereby meeting personalized data display and analysis requirements.

[0098] In some embodiments, the task information includes at least the connection information corresponding to the source database, the connection information corresponding to the target database, the field information of the target data, the data aggregation information corresponding to the target data, and the device information corresponding to the target data; configuring the task information corresponding to the data task includes: configuring the connection information corresponding to the source database and the connection information corresponding to the target database through the first interface; configuring the field information, data aggregation information, and device information corresponding to the target data through the second interface, thereby obtaining the data task.

[0099] The connection information for the source database, i.e., the information of the source database itself, can include table names, table types, etc. The field information for the target data can include the voltage, current, sensor temperature, etc., of the faulty battery device, and the vehicle speed, vehicle status, and cumulative mileage of the vehicle corresponding to the faulty battery device. The data aggregation information for the target data can include maximum, minimum, average, median, etc. Of course, the data aggregation information for the target data can also include values ​​obtained according to predefined formulas. The connection information for the target database can include the table type, table name, URL, etc., of the target data table. The device information for the target data can include the VIN (Vehicle Identification Number) ID of the faulty battery device and the data range corresponding to the faulty battery device. The data range corresponding to the faulty battery device represents the data within the time range of the faulty battery device's failure. The time range of the failure can be a period of time before the failure and a period of time after the failure. For example, within a month before and after the failure, the start time is one month before the failure and the end time is one month after the failure.

[0100] like Figure 6 As shown, the connection information for the source database and the target database can be configured through the first interface 61. For example, the table name and table type of the source database, as well as the table type, table name, and connection information of the target database table can be configured on the first interface 61.

[0101] The second interface 62 configures the field information, data aggregation information, and corresponding device information of the target data. For example, on the second interface 62, the total voltage or current, "maximum value," VIN ID of the faulty battery device, and the data range corresponding to the faulty battery device can be configured. The VIN ID of the faulty battery device can be manually imported or imported by uploading a file.

[0102] In the above solution, the connection information of the source database and the target database are configured through the first interface, and the field information, data aggregation information and device information of the target data are configured through the second interface, thereby obtaining the data task and realizing the visualization configuration of the data task. It can dynamically generate customized data table structures according to the user's specific needs, so that different users can generate different data table structures, thereby meeting personalized data display and analysis needs.

[0103] In some embodiments, task information includes the display method of the target data; the task information corresponding to the configuration data task includes: the display method of the target data.

[0104] The way target data is displayed can be configured through the interface of electronic devices, for example, Figure 5 The first interface 61 or the second interface 62 can be used. The target data can be displayed directly on the interface. Alternatively, the target data can be processed first, such as by creating charts, and then displayed in the form of charts.

[0105] In the above solution, by configuring the display method of target data, the visualization configuration data task can be further realized. It can dynamically generate customized data table structures according to the specific needs of users, so as to generate different data table structures for different users, thereby meeting personalized data display and analysis needs.

[0106] In some embodiments, scanning a fault list to obtain a data task includes: scanning the fault list according to a preset period to obtain a data task.

[0107] The preset cycle can be set according to the user's needs. Once the fault list is scanned, the electronic device automatically acquires data.

[0108] In the above solution, by periodically scanning the fault list to obtain data tasks, customized data table structures can be dynamically generated according to the user's specific needs, so as to generate different data table structures for different users, thereby meeting personalized data display and analysis needs.

[0109] In some embodiments, the data processing method further includes: recording data tasks and their task status in a preset task list.

[0110] The preset task list is used to manage data tasks. For example, when a new data task is retrieved, a new record is added to the preset task list to represent the data task and record its task status.

[0111] In the above scheme, data tasks and their status are recorded in a preset task list, which enables efficient and flexible management of data tasks while maintaining effective control over the corresponding data.

[0112] In some embodiments, in response to acquiring a data task, the task state of the data task is a waiting state; in response to executing the data task, the task state of the data task is a started state; in response to canceling the data task, the task state of the data task is a canceled state.

[0113] When a data task is acquired, it is recorded in a preset task list and its task status is marked as WAITING. When the data task is executed, its task status is updated to RUNNING in the preset task list. When a data task is canceled, its task status is updated to CANCELED in the preset task list. For example, if the data task is in the WAITING state and is then canceled, its task status in the preset task list will change from WAITING to CANCELED. Whether a data task has been acquired can be determined through periodic scanning; if acquired, it is designated as a WAITING task.

[0114] The above solution achieves efficient and flexible management of data tasks by setting different task states for data tasks, while maintaining effective control over the corresponding data.

[0115] In some embodiments, recording data tasks and their task status in a preset task list includes: in response to executing a data task and successfully synchronizing target data from the source database to the target database, recording the task status of the data task as a success status in the preset task list; in response to executing a data task but failing to successfully synchronize target data from the source database to the target database within a preset time, recording the task status of the data task as a failure status in the preset task list.

[0116] When a data task is executed and successfully synchronizes the target data from the source database to the target database, the task status in the preset task list is updated from RUNNING to FINISHED. If a data task is executed but fails to synchronize the target data from the source database to the target database within a preset time (e.g., within 24 hours), the task status in the preset task list is updated from RUNNING to FAILED.

[0117] For data tasks in the startup state, a timed process can be executed to monitor their execution status. For example, the data tasks can be scanned at preset time intervals, and their real-time task status can be queried. When a data task is completed, its task status is updated from RUNNING to FINISHED in the preset task list. When a data task is not completed, its task status is updated from RUNNING to FAILED in the preset task list.

[0118] In the above scheme, when a data task is executed and the target data is successfully synchronized from the source database to the target database, the task status of the data task is recorded as a success in the preset task list. When a data task is executed but the target data is not successfully synchronized from the source database to the target database within a preset time, the task status of the data task is recorded as a failure in the preset task list. This achieves effective task monitoring, maintains the accuracy and timeliness of task status, and thus enables efficient and flexible management of data tasks while maintaining effective control over the corresponding data.

[0119] In some embodiments, recording data tasks and their task status in a preset task list further includes: updating the task status of the data task in the preset task list from a successful state to an expired state in response to a first preset time when the data task is in a successful state; and updating the task status of the data task in the preset task list from a failed state to an expired state in response to a second preset time when the data task is in a failed state; wherein the second preset time is less than the first preset time.

[0120] Both the first and second preset times can be any time exceeding a certain threshold, such as a period measured in months. For example, if a data task has been in a successful (FINISHED) state for more than 3 months, the task status in the preset task list will be updated from FINISH to EXPIRED. If a data task has been in a failed (FAILED) state for more than 1 month, the task status in the preset task list will be updated from Failed to EXPIRED.

[0121] In the above scheme, the task status of the data task is updated from successful to expired after a first preset time when the data task is in a successful state, and the task status of the data task is updated from failed to expired after a second preset time when the data task is in a failed state. This further enables effective task monitoring, maintains the accuracy and timeliness of task status, and thus achieves efficient and flexible management of data tasks while maintaining effective control over the corresponding data.

[0122] In some embodiments, the data processing method further includes: in response to a third preset time when the data task is in an expired state, deleting the target data table and / or the corresponding target data of the data task.

[0123] The third preset time can be set according to actual conditions. When a data task is in an expired state (EXPIRED for the third preset time), it indicates that the data task has become invalid. At this time, the target data table and / or the corresponding target data for the data task are deleted. In other words, for data tasks in an expired state (EXPIRED for the third preset time), the target data table and / or the corresponding target data for the data task are deleted. For data tasks in an expired state (EXPIRED), periodic scanning can be used to determine whether a particular data task in the expired state (EXPIRED) has become invalid.

[0124] In the above scheme, by responding to the third preset time when the data task is in an expired state, the target data table and / or the corresponding target data corresponding to the data task are deleted, so as to keep the target database clean and efficient, effectively utilize storage space, and realize timely data updates and effective management.

[0125] In some embodiments, such as Figure 7 As shown, the data processing flow includes the following steps:

[0126] Step S71: Data Acquisition Task.

[0127] Data tasks can be obtained through configuration or updates, or by scanning a fault list. In the configuration method, such as... Figure 6 As shown, the connection information for the source database and the target database is configured through the first interface 61. For example, the table name and table type of the source database, and the table type, table name, and URL of the target database table are configured on the first interface 61. The field information, data aggregation information, and device information corresponding to the target data are configured through the second interface 62. In the second interface 62, for example... Figure 8As shown, the first page 81 includes several configuration items: application level, original fields, chart name, chart type, and aggregation method. The application level represents the measurement points on the faulty battery device corresponding to the data task. All data in the source database is obtained from these measurement points. The original fields represent the field information of the target data corresponding to the data task. The chart name and chart type represent the display method corresponding to the data task, and the aggregation method represents the data aggregation information corresponding to the data task. The second page 82 includes several configuration items: data range and VIN ID. The data range represents the data time range of the faulty battery device corresponding to the data task, i.e., all data within the data time range to be synchronized by the data task. The VIN ID represents the device information corresponding to the data task. On the first page 81, the application level, original fields, chart name, chart type, and aggregation method are configured respectively. Subsequently, on the second page 82, the data range and VIN ID are configured respectively. After clicking "OK," the task information corresponding to the data task is obtained, i.e., the data task is generated.

[0128] In the update method, the data tasks in the preset task list correspond to the update controls. When the update control is clicked, a new data task will be generated.

[0129] In the fault list scanning method, the fault list is scanned, and incremental fault data is pulled from the corresponding database, such as fault warning information and status information of vehicles or faulty battery devices, thereby automatically generating data tasks.

[0130] Step S72: Scan data task.

[0131] After acquiring a data task, a new record is added to the preset task list to represent that data task. For example, Figure 9 As shown, the task status of this data task is WAITING. When the data task is canceled, its task status is updated to CANCELED in the preset task list. When the data task is submitted to the server, its task status is updated to RUNNING in the preset task list.

[0132] The server has a maximum limit (max) for parallel task execution, and a set number (num) of data tasks currently in the RUNNING state. Therefore, max-num data tasks in the WAITING state are submitted to the server. After submission, the task status of these max-num data tasks in the preset task list is updated from WAITING to RUNNING. At this point, the server configures corresponding task IDs for these RUNNING data tasks so that the server can manage them. When the server encounters a data task, it executes that data task.

[0133] The processing end can periodically scan all data tasks that are in the running state on the server and query their real-time task status on the server.

[0134] Specifically, when a data task is executed and completed, such as Figure 8 As shown, the task status of a data task in the preset task list is updated from RUNNING to FINISHED. If a data task is executed but not completed within a preset time (e.g., within 24 hours), its task status is updated from RUNNING to FAILED. The processing end can then periodically scan the server for all data tasks in the RUNNING state to obtain the aforementioned task status.

[0135] When a data task is in a successful FINISHED state for a first preset time, for example, more than 3 months, the task status in the preset task list is updated from FINISHED to EXPIRED. When a data task is in a failed FAILED state for a second preset time, for example, more than 1 month, the task status in the preset task list is updated from failed to EXPIRED. Furthermore, if a data task is in an EXPIRED state for a third preset time, it indicates that the data task has expired. At this point, the target data table and / or the corresponding target data for the data task are deleted. In other words, for a data task in an EXPIRED state for a third preset time, the target data table and / or the corresponding target data for the data task are deleted.

[0136] Step S73: Execute the data task.

[0137] When a data task is executed, its task status is updated to RUNNING in the preset task list. During the execution of the data task, the processing end encodes the task information corresponding to the data task to obtain encoded task information, for example, in base64 format, and submits the encoded task information and the associated code identifier (code) to the server. The encoded task information (base64 code) and the associated code identifier (code) are as follows: Figure 10 As shown. It should be noted that... Figure 10 The information in the base64 code shown is for illustrative purposes only and represents the task information corresponding to the data task to be used in this application.

[0138] After receiving the encoding task information (base64 code) and the associated code identifier, the server queries the Spark stream or Spark task corresponding to the code identifier and triggers its execution. Specifically, upon receiving the encoding task information (base64 code) and the associated code identifier, the server executes the data task according to its corresponding task ID. The processing end can query the task status of the data task using that task ID.

[0139] During the execution of a Spark stream or Spark task, the encoded task information (base64 encoding) is first parsed to obtain the corresponding task information. This parsed encoded task information, i.e., the task details, can be in JSON format. Next, a connection is established with the source database according to the connection information. Then, based on the target data's field information, the raw data is obtained from the source database. Subsequently, aggregation operations are performed on the raw data according to the data aggregation information to obtain the target data. Finally, based on the target database's connection information, the target data is written to the target data table in the target database.

[0140] Among them, when the data task is executed and completed, such as Figure 9 As shown, the task status of a data task in the preset task list is updated from RUNNING to FINISHED. If a data task is executed but not completed within a preset time (e.g., within 24 hours), its task status is updated from RUNNING to FAILED. The processing end can then periodically scan the server for all data tasks in the RUNNING state to obtain the aforementioned task status.

[0141] Step S74: Query data.

[0142] After the data task is executed, the corresponding target data is written to the target data table in the target database. Different target data tables in the target database correspond to different execution cycles of the data task; that is, the data task is executed once per cycle, and the corresponding target data is synchronized to the corresponding target data table. For example, target data table 'a' corresponds to execution cycle 'b', and every time data task 'b' is executed, the corresponding target data is synchronized to target table 'a'. The execution cycle can be referred to as the time granularity; that is, different target data tables correspond to different time granularities. The different execution cycles of the data task are within the data time range of the faulty battery device, for example, within a preset time before and after the faulty battery device malfunctions.

[0143] Based on the interface display requirements, the system switches between querying at least one target data table to obtain the corresponding data; and switches between displaying at least one target data table on the interface to show the corresponding data. The data obtained from querying a target data table can be all the data in that target data table or a portion of the data in that target data table.

[0144] Steps S71-S73 and the source database described above can be deployed on the client side and can be run in a cluster or independently.

[0145] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of a data processing system provided in some embodiments of this application. The data processing system 110 includes a task acquisition module 111 and a task execution module 112. The task acquisition module 111 is used to acquire data tasks, wherein the data task represents a task to synchronize target data from a source database to a target database. The task execution module 112 is used to execute the data task to synchronize the target data from the source database to the target database; wherein the data volume of the target data table in the target database is smaller than the data volume of the source data table in the source database.

[0146] In the above solution, data acquisition tasks are performed to synchronize target data from the source database to the target database. The target database contains fewer data tables than the source database contains tables, allowing for the acquisition of small datasets as individual tasks, eliminating the need to read the entire dataset. This enables rapid adaptation, improves data query speed, and reduces storage space. Furthermore, synchronizing target data from the source database to the target database through data acquisition tasks allows for flexible database switching and enhances database adaptability.

[0147] In some embodiments, the task execution module 112 is configured to: encode the task information corresponding to the data task to obtain encoded task information; obtain the code identifier associated with the encoded task information; and process the encoded task information and the code identifier to synchronize the target data from the source database to the target database.

[0148] In the above scheme, the task information corresponding to the data task is encoded to obtain the encoded task information, and the code identifier associated with the encoded task information is obtained. Then, the data is processed according to the encoded task information and the code identifier to synchronize the target data from the source database to the target database. The database type can be switched at any time, which solves the problem of database type limitation in the battery CT scenario, thereby realizing flexible database switching and improving database adaptability.

[0149] In some embodiments, the processing based on the encoding task information and the code identifier includes: obtaining a preset task corresponding to the code identifier based on the code identifier; and executing the preset task to synchronize the target data from the source database to the target database based on the encoding task information.

[0150] In the above solution, by obtaining the preset task corresponding to the code identifier based on the code identifier, and then executing the preset task, the target data is synchronized from the source database to the target database according to the encoded task information. The database type can be switched at any time, solving the problem of database type limitation in the battery CT scenario, thereby realizing flexible database switching and improving database adaptability.

[0151] In some embodiments, synchronizing the target data from the source database to the target database based on the encoding task information includes: parsing the encoding task information to obtain task information, wherein the task information includes at least connection information corresponding to the source database, field information of the target data, data aggregation information, and connection information corresponding to the target database; establishing a connection with the source database according to the connection information corresponding to the source database; obtaining raw data from the source database according to the field information of the target data; performing an aggregation operation on the raw data according to the data aggregation information to obtain the target data; and writing the target data into a target data table in the target database according to the connection information corresponding to the target database.

[0152] In the above solution, by parsing the encoding task information, the task information can be obtained, enabling the switching of database types at any time. This solves the limitation of database types in battery CT scenarios, thereby achieving flexible database switching and improving database adaptability. Simultaneously, by writing target data into the target data table of the target database based on the task information, automated data processing is achieved, maintaining efficiency and reliability while improving data security and accuracy.

[0153] In some embodiments, the data processing system further includes a display module (not shown in the figure) for displaying the target data on a preset interface.

[0154] In the above solution, data visualization is achieved by displaying the target data on a preset interface, thus meeting the data visualization requirements.

[0155] In some embodiments, the target database includes at least one target data table, wherein the at least one target data table corresponds to different execution cycles of the data task.

[0156] In the above scheme, different data requirements are met by using at least one target data table in the target database to correspond to different execution cycles of the data task.

[0157] In some embodiments, the display module is configured to: switch between querying the at least one target data table to obtain the target data; and switch between displaying the at least one target data table on the preset interface to display the target data.

[0158] In the above solution, by switching to query at least one target data table and displaying at least one target data table on a preset interface, the target data is displayed. Since the amount of data in each target data table is less than the amount of data in the source data table of the source database, the computational burden of filtering redundant data and data aggregation operations is effectively reduced, thereby improving the speed of data query and the stability of the interface in the battery CT scenario, reducing the query load of the database, and improving system performance.

[0159] In some embodiments, the task acquisition module 111 is used to: configure and / or update the task information corresponding to the data task to obtain the data task; and / or scan the fault list to obtain the data task.

[0160] In the above solution, data tasks are obtained by configuring and / or updating the task information corresponding to the data tasks, and / or scanning the fault list. This allows for the acquisition of small-scale data on a single task basis, eliminating the need to read the entire dataset, enabling rapid adaptation, improving data query speed, and reducing storage space. Furthermore, by configuring and / or updating task information and / or scanning the fault list, customized data table structures can be dynamically generated according to the user's specific needs. This allows different users to generate different data table structures, thereby meeting personalized data display and analysis requirements.

[0161] In some embodiments, the task information includes at least the connection information corresponding to the source database, the connection information corresponding to the target database, the field information of the target data, the data aggregation information corresponding to the target data, and the device information corresponding to the target data;

[0162] The task acquisition module 111 is used to: configure the connection information corresponding to the source database and the connection information corresponding to the target database through a first interface; and configure the field information of the target data, the data aggregation information, and the device information corresponding to the target data through a second interface, thereby obtaining the data task.

[0163] In the above solution, the connection information of the source database and the target database are configured through the first interface, and the field information, data aggregation information and device information of the target data are configured through the second interface, thereby obtaining the data task and realizing the visualization configuration of the data task. It can dynamically generate customized data table structures according to the user's specific needs, so that different users can generate different data table structures, thereby meeting personalized data display and analysis needs.

[0164] In some embodiments, the task information includes the display method of the target data; the task acquisition module 111 is used to configure the display method of the target data.

[0165] In the above solution, by configuring the display method of target data, the visualization configuration data task can be further realized. It can dynamically generate customized data table structures according to the specific needs of users, so as to generate different data table structures for different users, thereby meeting personalized data display and analysis needs.

[0166] In some embodiments, the task acquisition module 111 is used to scan the fault list according to a preset period to obtain the data task.

[0167] In the above solution, by periodically scanning the fault list to obtain data tasks, customized data table structures can be dynamically generated according to the user's specific needs, so as to generate different data table structures for different users, thereby meeting personalized data display and analysis needs.

[0168] In some embodiments, the data processing system further includes a task recording module (not shown in the figure), used to record the data tasks and their task status in a preset task list.

[0169] In the above scheme, data tasks and their status are recorded in a preset task list, which enables efficient and flexible management of data tasks while maintaining effective control over the corresponding data.

[0170] In some embodiments, in response to acquiring the data task, the task state of the data task is a waiting state; in response to executing the data task, the task state of the data task is a start state; in response to canceling the data task, the task state of the data task is a canceled state.

[0171] The above solution achieves efficient and flexible management of data tasks by setting different task states for data tasks, while maintaining effective control over the corresponding data.

[0172] In some embodiments, the task recording module is configured to: in response to executing the data task and completing the synchronization of the target data from the source database to the target database, record the task status of the data task as a success status in the preset task list; in response to executing the data task and failing to complete the synchronization of the target data from the source database to the target database within a preset time, record the task status of the data task as a failure status in the preset task list.

[0173] In the above scheme, when a data task is executed and the target data is successfully synchronized from the source database to the target database, the task status of the data task is recorded as a success in the preset task list. When a data task is executed but the target data is not successfully synchronized from the source database to the target database within a preset time, the task status of the data task is recorded as a failure in the preset task list. This achieves effective task monitoring, maintains the accuracy and timeliness of task status, and thus enables efficient and flexible management of data tasks while maintaining effective control over the corresponding data.

[0174] In some embodiments, the task recording module is further configured to: update the task status of the data task from a successful state to an expired state in the preset task list in response to a first preset time when the data task is in a successful state; and update the task status of the data task from the failed state to an expired state in the preset task list in response to a second preset time when the data task is in a failed state; wherein the second preset time is less than the first preset time.

[0175] In the above scheme, the task status of the data task is updated from successful to expired after a first preset time when the data task is in a successful state, and the task status of the data task is updated from failed to expired after a second preset time when the data task is in a failed state. This further enables effective task monitoring, maintains the accuracy and timeliness of task status, and thus achieves efficient and flexible management of data tasks while maintaining effective control over the corresponding data.

[0176] In some embodiments, the data processing system further includes a data deletion module (not shown in the figure), configured to: delete the target data table and / or the corresponding target data corresponding to the data task in response to the data task being in an expired state for a third preset time.

[0177] In the above scheme, by responding to the third preset time when the data task is in an expired state, the target data table and / or the corresponding target data corresponding to the data task are deleted, so as to keep the target database clean and efficient, effectively utilize storage space, and realize timely data updates and effective management.

[0178] Please see Figure 12 , Figure 12 This is a schematic diagram of the structure of an electronic device provided in some embodiments of this application. The electronic device 120 includes a memory 121 and a processor 122. The processor 122 is used to execute program instructions stored in the memory 121 to implement the steps in any of the above data processing method embodiments. In a specific implementation scenario, the electronic device 120 may include, but is not limited to, a microcomputer or a server. Furthermore, the electronic device 120 may also include a laptop computer, tablet computer, or other carrier device, which is not limited here.

[0179] Specifically, processor 122 controls itself and memory 121 to implement the steps in any of the above data processing method embodiments. Processor 122 may also be referred to as a CPU (Central Processing Unit). Processor 122 may be an integrated circuit chip with signal processing capabilities. Processor 122 may also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 122 may be implemented using integrated circuit chips.

[0180] In the above solution, data acquisition tasks are performed to synchronize target data from the source database to the target database. The target database contains fewer data tables than the source database contains tables, allowing for the acquisition of small datasets as individual tasks, eliminating the need to read the entire dataset. This enables rapid adaptation, improves data query speed, and reduces storage space. Furthermore, synchronizing target data from the source database to the target database through data acquisition tasks allows for flexible database switching and enhances database adaptability.

[0181] Please see Figure 13 , Figure 13 This is a schematic diagram of the structure of a computer-readable storage medium provided in some embodiments of this application. The computer-readable storage medium 130 stores program instructions 1301 thereon, which, when executed by a processor, implement the steps in any of the above-described data processing method embodiments.

[0182] In the above solution, data acquisition tasks are performed to synchronize target data from the source database to the target database. The target database contains fewer data tables than the source database contains tables, allowing for the acquisition of small datasets as individual tasks, eliminating the need to read the entire dataset. This enables rapid adaptation, improves data query speed, and reduces storage space. Furthermore, synchronizing target data from the source database to the target database through data acquisition tasks allows for flexible database switching and enhances database adaptability.

[0183] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0184] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0185] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. In another image location, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0186] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A data processing method, characterized by, include: The data acquisition task, wherein the data task refers to the task of synchronizing target data from the source database to the target database; Execute the data task to synchronize the target data from the source database to the target database; The amount of data in the target data table in the target database is less than the amount of data in the source data table in the source database.

2. The data processing method according to claim 1, characterized in that, The execution of the data task includes: The task information corresponding to the data task is encoded to obtain encoded task information; Obtain the code identifier associated with the encoding task information; Based on the encoding task information and the code identifier, the target data is synchronized from the source database to the target database.

3. The data processing method according to claim 2, characterized in that, The processing based on the encoding task information and the code identifier includes: Based on the code identifier, obtain the preset task corresponding to the code identifier; The preset task is executed to synchronize the target data from the source database to the target database based on the encoded task information.

4. The data processing method according to claim 3, characterized in that, The step of synchronizing the target data from the source database to the target database based on the encoding task information includes: The encoding task information is parsed to obtain the task information, wherein the task information includes at least the connection information corresponding to the source database, the field information of the target data, the data aggregation information, and the connection information corresponding to the target database; Establish a connection with the source database according to the connection information corresponding to the source database; Based on the field information of the target data, the original data is obtained from the source database; The original data is aggregated according to the data aggregation information to obtain the target data; Based on the connection information corresponding to the target database, the target data is written into the target data table of the target database.

5. The data processing method according to any one of claims 1 to 4, characterized in that, Also includes: The target data is displayed on a preset interface.

6. The data processing method according to any one of claims 1 to 5, characterized in that, The target database includes at least one target data table, wherein the at least one target data table corresponds to different execution cycles of the data task.

7. The data processing method according to claim 6, characterized in that, The step of displaying the target data on a preset interface includes: Switch to querying at least one of the target data tables to obtain the target data; The at least one target data table is switched on the preset interface to display the target data.

8. The data processing method according to any one of claims 1 to 7, characterized in that, The data acquisition task includes: Configure and / or update the task information corresponding to the data task to obtain the data task; and / or, Scan the fault list to obtain the data task.

9. The data processing method according to claim 8, characterized in that, The task information includes at least the connection information corresponding to the source database, the connection information corresponding to the target database, the field information of the target data, the data aggregation information corresponding to the target data, and the device information corresponding to the target data; The task information corresponding to the configured data task includes: Configure the connection information for the source database and the connection information for the target database through the first interface; The data task is obtained by configuring the field information of the target data, the data aggregation information, and the device information corresponding to the target data through the second interface.

10. The data processing method according to claim 8 or 9, characterized in that, The task information includes the way the target data is displayed; The task information corresponding to the configured data task includes: Configure the display method of the target data.

11. The data processing method of claim 8, wherein, The task of scanning the fault list to obtain the data includes: The fault list is scanned according to a preset cycle to obtain the data task.

12. The data processing method according to any one of claims 1-11, characterized in that, Also includes: Record the data tasks and their status in the preset task list.

13. The data processing method of claim 11, wherein, In response to the acquisition of the data task, the task status of the data task is a waiting state; In response to the execution of the data task, the task status of the data task is in the started state; In response to the cancellation of the data task, the task status of the data task is cancelled.

14. The data processing method according to claim 12 or 13, characterized in that, The step of recording the data tasks and their status in the preset task list includes: In response to executing the data task and completing the synchronization of the target data from the source database to the target database, the task status of the data task is recorded as a success status in the preset task list; If the data task is not completed within a preset time and the target data is not synchronized from the source database to the target database, the task status of the data task is recorded as a failure in the preset task list.

15. The data processing method according to any one of claims 12-14, characterized in that, Recording the data tasks and their status in the preset task list also includes: In response to the data task being in a successful state for a first preset time, the task status of the data task in the preset task list is updated from successful state to expired state. In response to the data task being in a failed state for a second preset time, the task status of the data task in the preset task list is updated from the failed state to the expired state. The second preset time is shorter than the first preset time.

16. The data processing method according to any one of claims 12-14, characterized in that, Also includes: In response to the data task being in an expired state for a third preset time, the target data table and / or the corresponding target data corresponding to the data task are deleted.

17. A data processing system, characterized by include: The task acquisition module is used to acquire data tasks, wherein the data tasks refer to the task of synchronizing target data from the source database to the target database; The task execution module is used to execute the data task to synchronize the target data from the source database to the target database; The amount of data in the target data table in the target database is less than the amount of data in the source data table in the source database.

18. An electronic device, comprising: It includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the data processing method according to any one of claims 1 to 16.

19. A computer readable storage medium having stored thereon program instructions, wherein, When the program instructions are executed by the processor, they implement the data processing method according to any one of claims 1 to 16.

Images