Internet of vehicles data scheduling method and device, and related equipment

Abstract

The invention provides an Internet of Vehicles data scheduling method and device and related equipment, and the method comprises the steps: determining delay data and / or non-delay data in to-be-batch Internet of Vehicles data according to the library falling time of the data; time attributes of the delay data and the non-delay data are determined, according to the time attributes, aggregation processing and associated vehicle type processing are carried out on the delay data and / or the non-delay data, and event complete delay data of the associated vehicle type and / or event complete non-delay data of the associated vehicle type are / is determined; and based on preset data scheduling time, in combination with the time attributes of the delay data and / or the non-delay data, inserting the event complete delay data of the associated vehicle type and / or the complete non-delay data of the associated vehicle type into an updatable target table of the data lake so as to complete scheduling batch running of the event complete data. Different types of data are subjected to batch running by reasonably arranging batch running time, so that the batch running scheduling efficiency is improved, and the influence on other information transmission is reduced.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a method, device and related equipment for scheduling data of an Internet of Vehicles. Background Art

[0002] With the development of Internet of Vehicles technology, more and more data is transmitted in Internet of Vehicles applications, and the traffic is also increasing. Corresponding to the current Internet of Vehicles data application scenarios, Internet of Vehicles data processing conditions such as rapid calculation and processing of various event data generated during vehicle driving and flushing of historical event data are the prerequisites for efficient and accurate analysis of Internet of Vehicles data, data mining, vehicle simulation and other downstream data applications in the vehicle industry. With the rapid development of the smart car industry, convenient and efficient acquisition of complete data on the vehicle side has become an increasingly urgent technology for every automobile company. However, due to the complexity of different event data on the vehicle side, the huge amount of data, and data transmission delays, it is often difficult to ensure that the vehicle-side event data obtained by real-time batch running in vehicle work has good integrity and timeliness when using the vehicle-side data, and it is also difficult to ensure that the downstream industry can carry out development and research based on these data normally. In order to solve this problem, centralized batching of vehicle-side event data based on Internet of Vehicles technology has become the direction of data scheduling and batching in the industry. However, the traditional data scheduling and batching methods currently used in the industry mostly schedule batching according to daily data (one or more days of data), and the batching time is not arranged reasonably. At the same time, batching by day will also cause the way of writing data to be mostly in the form of overwriting the old data, which will occupy the data transmission channel or other hardware resources on the vehicle side during the data batching process, affecting the vehicle's user experience. The batching process also consumes a lot of energy, and the efficiency of data batching is difficult to guarantee. Therefore, how to provide a method for scheduling Internet of Vehicles data with a more reasonable batching time, less impact on the data transmission process of other functions on the vehicle side, and lower energy consumption has become a technical problem that needs to be solved urgently in the industry. Summary of the invention

[0003] In view of this, embodiments of the present application provide a method, device and storage medium for scheduling data in an Internet of Vehicles to at least partially solve the above-mentioned problems.

[0004] In a first aspect, an embodiment of the present application provides a method for scheduling data in an Internet of Vehicles, including:

[0005] Determine the delayed data and / or non-delayed data in the vehicle-side networked data to be batched based on the data arrival time;

[0006] Determine the time attributes of the delayed data and the non-delayed data in the vehicle-side network data to be run, wherein the time attributes at least include the date information and the corresponding hour information corresponding to the vehicle-side network data;

[0007] According to the time attribute, respectively, the delayed data and / or the non-delayed data are aggregated and associated with vehicle types to determine complete delayed event data of the associated vehicle type and / or complete non-delayed event data of the associated vehicle type;

[0008] Based on the preset data scheduling time, as well as the time attributes of each delayed data and non-delayed data, the target data is determined from the complete delayed data of the event and / or the complete non-delayed data of the event of the associated vehicle model, and the target data is inserted into the updateable target table of the data lake to complete the scheduling and batching of the vehicle-side networked data to be batched.

[0009] In a second aspect, based on the vehicle networking data scheduling method described in the first aspect of the present application, the present application further provides a vehicle networking data scheduling device, including:

[0010] An extraction module is used to determine delayed data and / or non-delayed data in the vehicle-side networked data to be batched based on the storage time of the data;

[0011] A determination module, used to determine the time attributes of delayed data and non-delayed data in the vehicle-side networked data to be run, wherein the time attributes at least include corresponding date information and corresponding hour information;

[0012] An association module, for performing data aggregation and vehicle model association processing on the delayed data and / or non-delayed data respectively according to the time attribute, and determining event complete delayed data of the associated vehicle model and / or event complete non-delayed data of the associated vehicle model;

[0013] A scheduling module is used to determine target data from the complete delayed data of the event and / or the complete non-delayed data of the event of the associated vehicle model based on the preset data scheduling time and the time attributes of each delayed data and non-delayed data, and insert the target data into an updateable target table of the data lake to complete the scheduling and batching of the vehicle-side networked data to be batched.

[0014] In a third aspect, an embodiment of the present application further provides a computer storage medium having computer executable instructions stored thereon, and when the computer executable instructions are executed, any one of the vehicle network data scheduling methods described in the first aspect of the present application is performed.

[0015] In a fourth aspect, an embodiment of the present application further provides an electronic device, including:

[0016] one or more processors;

[0017] a memory for storing one or more programs,

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement any one of the vehicle network data scheduling methods described in the first aspect of the present application.

[0019] The present application provides a method, device and related equipment for scheduling Internet of Vehicles data, which determines the delayed data and / or non-delayed data in the vehicle-side Internet of Vehicles data to be batched based on the preset data storage time; determines the time attributes of the delayed data and non-delayed data in the vehicle-side Internet of Vehicles data to be batched, and the time attributes at least include the corresponding date information and the corresponding hour information; according to the time attributes, respectively, the delayed data and / or non-delayed data are subjected to data aggregation and associated vehicle model processing to determine the complete delayed data of the associated vehicle model and / or the complete non-delayed data of the associated vehicle model; according to the preset data scheduling time, respectively, combined with the time attributes of the delayed data and / or non-delayed data, the complete delayed data of the associated vehicle model and / or the complete non-delayed data of the associated vehicle model are inserted into the updateable target table of the data lake to complete the scheduling and batching of the complete event data. The present application improves the user's vehicle experience by reasonably arranging the batching time, reducing the impact on the use of the Internet of Vehicles transmission channel for other functional data in daytime vehicle work. Based on the preset batch running time, different batch running strategies are formulated for the non-delayed data and delayed data corresponding to the preset batch running time to insert data into the target table of the target data lake, thereby reducing the amount of computation in the scheduling batch running process and improving the efficiency of data batch running. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0021] Figure 1 A schematic diagram of a workflow of a vehicle networking data scheduling method provided in an embodiment of the present application;

[0022] Figure 2 A schematic diagram of the structure of a vehicle networking data scheduling device provided in an embodiment of the present application;

[0023] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0025] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0026] Embodiment 1

[0027] The present application embodiment provides a method for scheduling data in an Internet of Vehicles. Figure 1 As shown, Figure 1 A flowchart of a vehicle networking data scheduling method provided in an embodiment of the present application, the vehicle networking data scheduling method includes:

[0028] Step S101, based on the storage time of the data, determine the delayed data and / or non-delayed data in the vehicle-side networked data to be batched. Among them, the delayed data is the vehicle-side networked data with the interval between the data generation time and the data storage time, and the non-delayed data is the vehicle-side networked data with the data generation time and the storage time connected in time. In the embodiment of the present application, based on the implementation time of the data, the delayed data and the non-delayed data are distinguished so that the two can be scheduled and batched in different ways in the future, which reduces the amount of data processing for the batching process of the vehicle-side networked data to be batched to a certain extent. In the embodiment of the present application, the storage time of the data is also called the processing time of the data in the industry. This data processing refers to the process of flink writing kafka data to hive at the minute level, that is, the time when the vehicle-side writes the generated vehicle network data such as business logs and business data to the local data warehouse hive. In preparation for the batch scheduling process, the event data is inserted or updated from the data warehouse to the target table of the data lake to complete the data batch scheduling process.

[0029] Optionally, in an implementation of the embodiment of the present application, based on the storage time of the data, the delayed data and / or non-delayed data in the vehicle-side network data to be batched are determined, including: querying the original table of event data of the vehicle-side network data to determine the vehicle-side network data to be batched, and according to the date parameter and hour parameter in the storage time of the data, determining the event data contained in the vehicle-side network data to be batched as delayed data if the generation date of the event data is different from the date parameter or the generation hour information of the event data is different from the hour parameter of the date parameter, and / or determining the event data of the vehicle-side network data to be batched as non-delayed data if the generation date of the event data is the same as the date parameter, and the generated hour information is the same as the hour parameter or the hour parameter contains the generated hour information. By querying the original table of event data of the vehicle-side network data, that is, describing the data generated by the vehicle-side to be dropped into the data table in the data warehouse hive, it is ensured that the event data occurring on the vehicle-side is accurately and quickly recorded and queried, and the comprehensiveness of the queried data that needs to be batched is ensured. Based on the data's arrival time, the data to be scheduled and batched are distinguished into delayed and non-delayed types, so that the vehicle-side networked data to be scheduled and batched can be classified and scheduled at the preset data scheduling time, so as to rationally utilize the batching time and system resources, and effectively ensure the integrity of the scheduling and batching of the vehicle network data.

[0030] Step S102, determine the time attributes of delayed data and non-delayed data in the vehicle-side networked data to be batched, and the time attributes at least include the date information and the corresponding hour information corresponding to the vehicle-side networked data. For example, when a user turns on an application of the vehicle while driving the vehicle-connected vehicle, the event data generated during the operation of the application sets the corresponding timestamp information, and the timestamp information includes the specific date information and hour information of the application being turned on, such as ABCD year Y month x day z o'clock e minute, the date information of the timestamp information is ABCD year Y month x day, and the hour information is z o'clock. The embodiment of the present application uses date information and hour information as the basis for scheduling and batching delayed data and non-delayed data, while ensuring the accuracy of the results of the scheduling and batching, and uses date days and hours as the basis for data scheduling and batching, so as to more reasonably schedule and batch the scheduling data.

[0031] Step S103, according to the time attribute, respectively perform aggregation processing and associated vehicle model processing on the delayed data and / or non-delayed data, and determine the complete delayed data of the events of the associated vehicle models and / or the complete non-delayed data of the events of the associated vehicle models. Wherein, in the embodiment of the present application, the aggregation processing at least includes extracting the target data, converting the data format, and merging the data based on event and time information, and after writing the processed Internet of Vehicles data into the target table of the data lake, it is ensured that the corresponding vehicle network data has better granularity and better readability. In the embodiment of the present application, according to the determined time attribute, the vehicle-side Internet of Vehicles data that needs to be scheduled and batched is associated based on the timeline and based on the time association describing the same work event, and the integrity of the data for scheduling and batching can also be ensured.

[0032] Step S104, based on the preset data scheduling time, and the time attributes of each delayed data and non-delayed data, determine the target data from the complete delayed data of the event and / or the complete non-delayed data of the event of the associated vehicle model, and insert the target data into the updateable target table of the data lake to complete the scheduling of the vehicle-side networked data to be run. In the embodiment of the present application, the target database is configured to exist in the form of a data lake, such as the data lake Hudi, which is an open source data lake architecture of Uber, and it is a streaming data lake built around the database kernel. The target table-COW table of the Hudi data lake mainly uses the columnar file format (Parquet) to store data. In the process of writing data, synchronous merging is performed, the data version is updated and the data file is rewritten, similar to the B-Tree update in RDBMS. Update update process: When updating a record, Hudi will first find the file containing the updated data, and then rewrite the file with the updated value (the latest data), and the file containing other records remains unchanged, avoiding the writing of data in an overwritten manner, reducing the resource consumption of data batch scheduling, and improving the efficiency of batch scheduling. Suitable for scenarios where a sudden large number of write operations will cause a large number of files to be rewritten, resulting in huge I / O overhead.

[0033] Optionally, in an implementation of the embodiment of the present application, the data scheduling method further includes setting the target table of the data lake to be updateable before step S104, so that when the target table is updated, the update is implemented through the assigned Key. In this way, when the delayed data is inserted into the database after batch scheduling, there is no need to perform window processing on the target table of the data lake database, but the data is stored in the target table of the database by insertion. The data writing process has low energy consumption and consumes less resources. While ensuring the accuracy of the updated data, it can also significantly improve the efficiency of writing data.

[0034] Optionally, in an implementation of the embodiment of the present application, the data scheduling method also includes, before step S104, configuring the loading class of the data lake target table to realize automatic merging and deduplication of the map when inserting data, while ensuring the post-insertion integrity of the event data associated with any event after insertion, and improving the efficiency of data insertion. Specifically, the embodiment of the present application is described here with the data lake hudi example. Specifically, the embodiment of the present application is still described here with the data lake Hudi as an example. In combination with the data lake target table, the process of automatically merging and deduplication of the map for data insertion by configuring the loading class is realized. By utilizing Apache Hudi's Payload, an extensible data processing mechanism, customized data writing methods for complex scenarios are realized through different Payloads, which greatly increases the flexibility of data processing. HudiPayload is a tool class that performs operations such as deduplication, filtering, and merging on data when writing and reading Hudi's target table. The Payload class to be used is specified by using the parameter "hoodie.datasource.write.payload.class". The hudi custom payload class OverwriteMapTypeWithLatestAvroPayload inherits the OverwriteWithLatestAvroPayload class, rewrites the combineAndGetUpdateValue() method, obtains all fields of the target data according to the schema.getFields() method, and determines whether field.schema() belongs to the Map type. If so, the data of the field is obtained and the two Map types of the newly inserted data are merged into new Map data by the putAll() method, and the currently returned Map type data is encapsulated into the current data. The new data returned is the merged and deduplicated data.

[0035] Optionally, in an embodiment of the present application, based on the preset data scheduling time, and the time attributes of each delayed data and non-delayed data, the target data is determined from the complete delayed data of the event and / or the complete non-delayed data of the event of the associated vehicle model, and the target data is inserted into the updateable target table of the data lake to complete the scheduling and batching of the vehicle-side networked data to be batched, including: for the complete non-delayed data of the associated vehicle model: if the preset data scheduling time is within the first time interval, the event data in the non-delayed data of the previous hour whose time attribute is within the upper and lower limits of the first time interval is determined as the target data, and the target data is inserted into the updateable target table of the data lake by incremental update to complete the scheduling and batching of the non-delayed data; if the preset data scheduling time is the second time interval, the event data in the non-delayed data whose time attribute is the same as the data batch date and whose hour information is between 1 o'clock and 5 o'clock is inserted into the updateable target table of the data lake by incremental update to complete the scheduling and batching of the non-delayed data. Specifically, in the example of the present application, the first time interval or the second time interval is set according to the time of the area where the vehicle end is located. In general, for non-delayed data, that is, event data generated in the system working on the vehicle side within the day or the current time period (hour information), it can be set according to the time when most vehicles are normally used in the time zone where the vehicle is located, so as to reasonably allocate the batch running time. Preferably, in an optional implementation method of the present application, the first time interval is set to between 7 o'clock every day and 1 o'clock the next night, so that the event data of the hour before the current corresponding hour in the time attribute of the non-delayed complete event data is determined as the target data, that is, the complete event data between 6 o'clock on the day and 1 o'clock on the next day is used as the Internet of Vehicles target data to be batched, and the Internet of Vehicles target data is inserted into the target table of the data lake by incremental update. In addition, the second time interval can also be set to 6:00 to 7:00 in the morning every day. This time period is when a certain but still small number of users use the vehicle. At this time, the event data with time attributes and the hour information of the event data on the same day in the non-delayed data from 1:00 to 5:00 is inserted into the updateable target table of the data lake by incremental update. The time period from 1:00 to 5:00 is the time when most users do not use their cars, which means that the amount of data generated by the vehicle side during this time period that needs to be scheduled and batched is relatively small. The network transmission channel corresponding to the time period when a small number of users use their cars is used to schedule and batch the relatively small amount of non-delayed data, so as to reasonably and fully utilize the batching resources. This can not only ensure the accuracy and completion rate of scheduling and batching the small amount of data within this time period, but also reduce the impact of the resources consumed in the scheduling and batching process on the work or communication of other systems on the vehicle side.Of course, the embodiment of the present application is only an illustrative description of the first time interval and the second time interval. Those skilled in the art can flexibly set them according to the needs of different work scenarios, and the embodiment of the present application does not limit this.

[0036] Optionally, in one embodiment of the present application, based on the preset data scheduling time and the time attributes of each delayed data and non-delayed data, the target data is determined from the event complete delayed data and / or the event complete non-delayed data of the associated vehicle model, and the target data is inserted into an updateable target table of the data lake to complete the scheduling and batching of the vehicle-side networked data to be batched, including: performing data splitting on the event complete delayed data of the associated vehicle model; calculating the number of storage partitions that need to be updated for the split data batching according to the splitting results; based on the preset data scheduling time, the number of storage partitions that need to be updated and the data generation time, the event complete delayed data of the associated vehicle model is inserted into the updateable target table of the data lake in a serial batching manner to complete the scheduling and batching of the delayed data.

[0037] Optionally, in one embodiment of the present application, the data splitting of the complete event delay data of the associated vehicle model includes: splitting the complete event delay data of the associated vehicle model according to the generation time of the event data contained in the delay data, and / or splitting the complete event delay data of the associated vehicle model according to the storage time of the data. In the embodiment of the present application, the data is split according to the generation time of the event data. In the process of writing to the target table, the amount of data read in each time is large, but the target table written to the data lake has fewer partitions that need to be updated as a whole. In the process of splitting the data according to the storage time of the data, in the process of writing to the target table of the database after scheduling and batching, more partitions need to be updated, but the amount of data that needs to be read each time is small. By splitting the delayed data differently according to the actual situation and then scheduling and batching, the efficiency of data scheduling and batching can be improved according to different situations.

[0038] Specifically, in one embodiment of the present application, based on the preset data scheduling time, the number of storage partitions that need to be updated and the generation time of the data, the event complete delay data of the associated vehicle model is inserted into the updateable target table of the data lake in a serial batch manner to complete the scheduling and batching of the delay data, including: if the splitting result of the data splitting of the event complete delay data of the associated vehicle model according to the storage time of the data calculates that the number of storage partitions that need to be updated for the split data batch is greater than or equal to a first preset threshold, then the event complete delay data of the associated vehicle model is split according to the generation time of the event data contained in the delay data to determine the first splitting result; if the preset data scheduling time starts at 0 o'clock, then the event complete delay data of the associated vehicle model is sorted in the order of the generation time of the event data, and combined with the first splitting result, the event complete delay data of the associated vehicle model is inserted into the updateable target table of the data lake in a serial batch manner to complete the scheduling and batching of the delay data, in an embodiment of the present application. In general, the batch scheduling data starting at 0 o'clock is specifically the delayed data of the previous day; if the preset data scheduling time starts at 1 o'clock, the complete delayed data of the events of the associated vehicle models is inserted into the updateable target table of the data lake in a serial batch mode to complete the scheduling of the delayed data. The batch scheduling data starting at 1 o'clock is the event data of 0 o'clock and later on the same day. In the embodiment of the present application, preferably, the first preset threshold is 2000. Specifically, in the actual application scenario of the embodiment of the present application, according to the experiment, after the delayed event data is split according to the data storage time, when it is determined that more than or equal to 2000 storage partitions need to be updated according to the split result, the partitions that need to be updated in the target table of the data lake after the event data is split according to the span hours of the event data are greater than or equal to 200 storage partitions. If it takes more than 9 hours to schedule the batch after the data is split according to the data storage time, in this case, the time required for the batch after the split according to the span hours of the event data is slightly more than two hours. The embodiment of the present application performs reasonable data splitting and scheduling in batches in this way, which can further improve the efficiency of data batch running and reduce the consumption of system resources.

[0039] If the result of data splitting of the complete event delay data of the associated vehicle model according to the data storage time calculates that the number of storage partitions that need to be updated for the split data batch is less than or equal to the second preset threshold, then the delayed data is split according to the duration of the data storage time, and the second splitting result is determined; if the preset data scheduling time starts at 0 o'clock, combined with the second splitting result, the complete event delay data of the associated vehicle model is sorted according to the generation time of the event data, and the complete event delay data of the associated vehicle model is inserted into the updateable target table of the data lake in a serial batch manner to complete the scheduling of the delay data; if the preset data scheduling time starts at 1 o'clock, the complete event delay data of the associated vehicle model with the data generation date of the day, that is, the delay data on and after 0 o'clock on the day, is inserted into the updateable target table of the data lake in a serial batch manner to complete the scheduling of the delay data.

[0040] Preferably, in the embodiment of the present application, the second preset threshold is 1000. Specifically, in the actual application scenario of the embodiment of the present application, according to the experiment, after the delayed event data is split according to the duration of the data's storage time, when it is determined according to the split result that more than or equal to 1000 storage partitions need to be updated, such as when slightly more than 1000 storage partitions, in this case, the time required for running the batch after splitting according to the span of the event data in hours is slightly more than two hours. And the partitions that need to be updated in the target table of the data lake after splitting according to the span of the event data are greater than or equal to 200 storage partitions. If the time for scheduling the batch after splitting the data according to the duration of the data's storage time is less than two hours, the scheduling batch speed can be slightly faster. The embodiment of the present application schedules the batch after reasonable data splitting in this way, which can further improve the efficiency of data batch running and reduce the consumption of system resources. And by arranging the scheduling and batching time of the complete event delay data of the associated vehicles at midnight or early morning, the resource preemption pressure of the scheduling and batching can be further reduced, and the resource utilization rate of the daytime hourly batching of non-delayed data can be improved to a certain extent. Set the preset data scheduling time to 0 o'clock, then the event data whose date in the time attribute of the event data is not the same day, or the hour information in the event data and the storage time are different hours, such as the delay data with hour information of 0 to 9 o'clock on the previous day (yesterday) in the vehicle-side network data, is scheduled and batched, and then the delay data with hour information of 10 to 11 o'clock is scheduled and batched, until the delay data of yesterday with hour attribute of 23 o'clock is finally scheduled and batched. If the preset data scheduling time starts at 1 o'clock, the date information of the event data and the date of the storage time are different days (dates), or the hour information of the event data and the hour information of the storage time are different hours, are scheduled and batched.

[0041] Furthermore, in the above-mentioned implementation method of the embodiment of the present application, at midnight or early morning, such as starting at zero o'clock to schedule and batch the non-delayed data and delayed data corresponding to that moment, it is also possible to reduce the complexity of data processing when batching event data spanning several days and dates, thereby ensuring the accuracy of data scheduling and batching.

[0042] Optionally, in an implementation of an embodiment of the present application, the data scheduling method further includes: after the delayed data of the previous day is batched, a batch completion mark corresponding to the previous day's data is written in the corresponding completion position of the target table, which is used to indicate that yesterday's data or the delayed data corresponding to 0 o'clock on the current day has been batched, so as to improve the accuracy of the batching through this mark.

[0043] Optionally, in one embodiment of the present application, the data scheduling method further includes: setting a non-scheduled time interval for batching event data that failed to be batched on other dates and delayed data that falls within the non-scheduled time interval. Specifically, in an implementation scenario of an embodiment of the present application, the time interval between 2 a.m. and 5 a.m. every day can be set as a non-scheduled time interval to provide a suitable time window for the batching process of data that failed to be batched on other dates or delayed data on the day itself, to perform scheduling and batching, and further improve the scheduling accuracy of the embodiment of the present application.

[0044] Optionally, in one embodiment of the present application, the method further includes: dynamically allocating data processing resources and / or network transmission resources that can be used by the data batch process within the preset data scheduling time. For example, a network transmission threshold for scheduling batches is set during the time period when most users use their cars during the day to prevent data batches from occupying too many system resources. At the same time, the parallelism of inserting and updating the data lake target table can also be set to improve the efficiency of data scheduling batches by increasing the number of scheduling tasks and reducing the amount of data for each scheduling task.

[0045] The present application provides a method for scheduling Internet of Vehicles data, which determines the delayed data and / or non-delayed data in the vehicle-side Internet of Vehicles data to be batched based on the data storage time; determines the time attributes of the delayed data and non-delayed data in the vehicle-side Internet of Vehicles data to be batched, and the time attributes at least include the corresponding date information and the corresponding hour information; according to the time attributes, respectively, the delayed data and / or non-delayed data are subjected to data aggregation and associated vehicle model processing to determine the complete delayed data of the associated vehicle model and / or the complete non-delayed data of the associated vehicle model; according to the preset data scheduling time, respectively, combined with the time attributes of the delayed data and / or non-delayed data, the complete delayed data of the associated vehicle model and / or the complete non-delayed data of the associated vehicle model are inserted into the updateable target table of the data lake to complete the scheduling and batching of the complete event data. The present application improves the user's vehicle experience by reasonably arranging the batching time, reducing the impact on the use of the Internet of Vehicles transmission channel for other functional data in daytime vehicle work. At the preset data scheduling time, different batch running strategies are formulated for the non-delayed data and delayed data corresponding to the preset batch running time to insert data into the target table of the target data lake, thereby reducing the amount of calculation in the scheduling batch running process and improving the efficiency of data batch running.

[0046] Embodiment 2

[0047] Based on the vehicle networking data scheduling method described in the first embodiment of the present application, the present application embodiment further provides a vehicle networking data scheduling method 20, such as Figure 2 As shown, Figure 2 The present invention provides a schematic diagram of a vehicle networking data scheduling device 20 according to an embodiment of the present invention. The vehicle networking data scheduling device 20 includes:

[0048] Extraction module 201, used to determine delayed data and / or non-delayed data in the vehicle-side networked data to be batched based on the storage time of the data;

[0049] A determination module 202 is used to determine the time attributes of delayed data and non-delayed data in the vehicle-side networked data to be run, and the time attributes at least include corresponding date information and corresponding hour information;

[0050] The association module 203 is used to perform data aggregation and vehicle model association processing on the delayed data and / or non-delayed data according to the time attribute, and determine the complete delayed event data of the associated vehicle model and / or the complete non-delayed event data of the associated vehicle model;

[0051] The scheduling module 204 is used to determine the target data from the complete delayed data of the event and / or the complete non-delayed data of the event of the associated vehicle model based on the preset data scheduling time and the time attributes of each delayed data and non-delayed data, and insert the target data into the updateable target table of the data lake to complete the scheduling and batching of the vehicle-side networked data to be batched.

[0052] Optionally, in an implementation of the embodiment of the present application, the extraction module 201 is further used to:

[0053] Query the original table of event data of Internet of Vehicles data to determine the Internet of Vehicles data to be batched;

[0054] According to the date parameter and hour parameter in the preset data scheduling time, the event data in the Internet of Vehicles data to be run whose generation date is different from the date parameter or whose generation hour information of the event data is different from the hour parameter of the date parameter is determined as delayed data, and / or, the event data in the Internet of Vehicles data to be run whose generation date is the same as the date parameter and whose generated hour information is the same as the hour parameter or whose hour parameter includes the generated hour information is determined as non-delayed data.

[0055] Optionally, in an implementation of the embodiment of the present application, the scheduling module 204 is further configured to:

[0056] For complete non-delayed data on the associated vehicle models:

[0057] If the preset data scheduling time is within the first time interval, the event data in the non-delayed data of the previous hour whose time attribute is within the upper limit and the lower limit of the first time interval is inserted into the updateable target table of the data lake by incremental update, so as to complete the scheduling of the non-delayed data;

[0058] If the preset data scheduling time is the second time interval, the event data in the non-delayed data whose time attribute is the same as the data batch date and whose hour information is between 1:00 and 5:00 is inserted into the updateable target table of the data lake through incremental update to complete the scheduling of the non-delayed data.

[0059] Optionally, in an implementation of the embodiment of the present application, the scheduling module 204 is further configured to:

[0060] Performing data splitting on the complete event delay data of the associated vehicle model;

[0061] Calculate the number of storage partitions that need to be updated for the split data batch based on the split results;

[0062] Based on the preset data scheduling time, the number of storage partitions that need to be updated and the data generation time, the complete delayed data of the events of the associated vehicle models are inserted into the updateable target table of the data lake in a serial batch manner to complete the scheduling of the delayed data.

[0063] Optionally, in an implementation of the embodiment of the present application, the scheduling module 204 is further configured to:

[0064] The complete event delay data of the associated vehicle model is split according to the generation time of the event data contained in the delay data, and / or the complete event delay data of the associated vehicle model is split according to the duration of the data's arrival in the warehouse.

[0065] Optionally, in an implementation of the embodiment of the present application, the scheduling module 204 is further configured to:

[0066] If the number of storage partitions that need to be updated for the data batch after the splitting result of the data splitting of the event complete delay data of the associated vehicle model according to the length of the data storage time is greater than or equal to the first preset threshold, the event complete delay data of the associated vehicle model is split according to the generation time of the event data contained in the delay data to determine the first splitting result;

[0067] If the preset data scheduling time starts at 0 o'clock, the complete event delay data of the associated vehicle model is sorted according to the generation time sequence of the event data, and combined with the first split result, the complete event delay data of the associated vehicle model is inserted into the updateable target table of the data lake in a serial batch manner to complete the scheduling of the delay data; if the preset data scheduling time starts at 1 o'clock, the complete event delay data of the associated vehicle model is inserted into the updateable target table of the data lake in a serial batch manner to complete the scheduling of the delay data;

[0068] If the number of storage partitions that need to be updated for the split data batch after the split is calculated by performing data splitting on the complete delayed event data of the associated vehicle model according to the length of the data's storage time is less than or equal to a second preset threshold, then after performing data splitting on the delayed data according to the length of the data's storage time, determine a second splitting result;

[0069] If the preset data scheduling time starts at 0 o'clock, combined with the second splitting result, the event complete delay data of the associated vehicle model is sorted in the order of the generation time of the event data, and the event complete delay data of the associated vehicle model is inserted into the updateable target table of the data lake in a serial batch manner to complete the scheduling and batching of the delay data; if the preset data scheduling time starts at 1 o'clock, the event complete delay data of the associated vehicle model with the data generation date of the day is inserted into the updateable target table of the data lake in a serial batch manner to complete the scheduling and batching of the delay data.

[0070] Optionally, in one implementation of the embodiment of the present application, the Internet of Vehicles data scheduling device 20 also includes a non-scheduling module (not shown in the drawings), which is used to set a non-scheduling time interval for batching event data that failed to be batched on other dates and delayed data that falls within the non-scheduling time interval.

[0071] Optionally, in one implementation of the embodiment of the present application, the Internet of Vehicles data scheduling device 20 also includes an allocation module (not shown in the drawings), which is used to dynamically allocate data processing resources and / or network transmission resources that can be used in the data batch process within the preset data scheduling time.

[0072] The present application provides a vehicle networking device, which determines the delayed data and / or non-delayed data in the vehicle-side networked data to be batched based on the storage time of the data through an extraction module; sets a determination module to determine the time attributes of the delayed data and non-delayed data in the vehicle-side networked data to be batched, and the time attributes at least include corresponding date information and corresponding hour information; sets an association module to perform data aggregation and associated vehicle model processing on the delayed data and / or non-delayed data according to the time attributes, and determines the complete delayed data of the associated vehicle model and / or the complete non-delayed data of the associated vehicle model; sets a scheduling module to insert the complete delayed data of the associated vehicle model and / or the complete non-delayed data of the associated vehicle model into the updateable target table of the data lake according to the preset data scheduling time, in combination with the time attributes of the delayed data and / or non-delayed data, so as to complete the scheduling and batching of the complete event data. The present application improves the user's car experience by reasonably arranging the batching time, reducing the impact of the use of the vehicle networking transmission channel on other functional data in daytime vehicle work. Based on the preset data scheduling time, different batch running strategies are formulated for the non-delayed data and delayed data corresponding to the time to insert data into the target table of the target data lake, thereby reducing the amount of calculation in the scheduling batch running process and improving the efficiency of data batch running.

[0073] Embodiment 3

[0074] The embodiment of the present application further provides a storage medium on which a computer program is stored. When the program is executed by a processor, any one of the vehicle networking data scheduling methods described in the first embodiment of the present application is implemented. The vehicle networking data scheduling method is not limited to:

[0075] Determine the delayed data and / or non-delayed data in the vehicle-side networked data to be batched based on the data arrival time;

[0076] Determine the time attributes of the delayed data and the non-delayed data in the vehicle-side network data to be run, wherein the time attributes at least include the date information and the corresponding hour information corresponding to the vehicle-side network data;

[0077] According to the time attribute, respectively, the delayed data and / or the non-delayed data are aggregated and associated with vehicle types to determine complete delayed event data of the associated vehicle type and / or complete non-delayed event data of the associated vehicle type;

[0078] Based on the preset data scheduling time, as well as the time attributes of each delayed data and non-delayed data, the target data is determined from the complete delayed data of the event and / or the complete non-delayed data of the event of the associated vehicle model, and the target data is inserted into an updateable target table of the data lake to complete the scheduling and batching of the vehicle-side networked data to be batched.

[0079] Embodiment 4:

[0080] The present application also provides an electronic device, such as Figure 3 As shown, the electronic device includes:

[0081] One or more processors 301;

[0082] The memory 303 is used to store one or more programs.

[0083] When the one or more programs are instructed to be executed by the one or more processors 301 through the communication bus 304 based on the operation command received through the communication interface 302, the one or more processors 301 implement the vehicle networking data scheduling method as described in any one of the above embodiments. The vehicle networking data scheduling method includes but is not limited to:

[0084] Determine the delayed data and / or non-delayed data in the vehicle-side networked data to be batched based on the data arrival time;

[0085] Determine the time attributes of the delayed data and the non-delayed data in the vehicle-side network data to be run, wherein the time attributes at least include the date information and the corresponding hour information corresponding to the vehicle-side network data;

[0086] According to the time attribute, respectively, the delayed data and / or the non-delayed data are aggregated and associated with vehicle types to determine complete delayed event data of the associated vehicle type and / or complete non-delayed event data of the associated vehicle type;

[0087] Based on the preset data scheduling time, as well as the time attributes of each delayed data and non-delayed data, the target data is determined from the complete delayed data of the event and / or the complete non-delayed data of the event of the associated vehicle model, and the target data is inserted into an updateable target table of the data lake to complete the scheduling and batching of the vehicle-side networked data to be batched.

[0088] So far, the application has described specific embodiments of the subject matter. Other embodiments are within the scope of the appended claims. In some cases, the actions recorded in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing can be advantageous.

[0089] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system layer on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0090] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0091] The system layers, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0092] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0093] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, system layers, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0094] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0095] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific transactions or implement specific abstract data types. The present application may also be practiced in distributed computing environments where transactions are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0096] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system layer embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0097] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for scheduling data in an Internet of Vehicles, characterized in that: include: Determine the delayed data and / or non-delayed data in the vehicle-side networked data to be batched based on the data arrival time; Determine the time attributes of the delayed data and the non-delayed data in the vehicle-side network data to be run, wherein the time attributes at least include the date information and the corresponding hour information corresponding to the vehicle-side network data; According to the time attribute, respectively, the delayed data and / or the non-delayed data are aggregated and associated with vehicle types to determine complete delayed event data of the associated vehicle type and / or complete non-delayed event data of the associated vehicle type; Based on the preset data scheduling time, as well as the time attributes of each delayed data and non-delayed data, the target data is determined from the complete delayed data of the event and / or the complete non-delayed data of the event of the associated vehicle model, and the target data is inserted into the updateable target table of the data lake to complete the scheduling and batching of the vehicle-side networked data to be batched.

2. The vehicle networking data scheduling method according to claim 1, characterized in that: The method of determining delayed data and / or non-delayed data in the vehicle-side networked data to be batched based on the data arrival time includes: Query the original table of event data of Internet of Vehicles data to determine the Internet of Vehicles data to be batched; According to the date parameter and hour parameter in the storage time of the data, the event data in the Internet of Vehicles data to be run whose generation date is different from the date parameter or whose generation hour information of the event data is different from the hour parameter of the date parameter is determined as delayed data, and / or, the event data in the Internet of Vehicles data to be run whose generation date is the same as the date parameter and whose generated hour information is the same as the hour parameter or whose hour parameter includes the generated hour information is determined as non-delayed data.

3. The vehicle networking data scheduling method according to claim 1, characterized in that: Based on the preset data scheduling time and the time attributes of each delayed data and non-delayed data, the target data is determined from the complete delayed event data and / or the complete non-delayed event data of the associated vehicle model, and the target data is inserted into an updateable target table of the data lake to complete the scheduling and batching of the vehicle-side networked data to be batched, including: For complete non-delayed data on the associated vehicle models: If the preset data scheduling time is within the first time interval, the event data in the non-delayed data of the previous hour whose time attribute is within the upper limit and the lower limit of the first time interval is inserted into the updateable target table of the data lake by incremental update, so as to complete the scheduling of the non-delayed data; If the preset data scheduling time is the second time interval, the event data in the non-delayed data whose time attribute is the same as the data batch date and whose hour information is between 1:00 and 5:00 is inserted into the updateable target table of the data lake through incremental update to complete the scheduling of the non-delayed data.

4. The vehicle networking data scheduling method according to claim 1, characterized in that: Based on the preset data scheduling time and the time attributes of each delayed data and non-delayed data, the target data is determined from the complete delayed event data and / or the complete non-delayed event data of the associated vehicle model, and the target data is inserted into an updateable target table of the data lake to complete the scheduling and batching of the vehicle-side networked data to be batched, including: Performing data splitting on the complete event delay data of the associated vehicle model; Calculate the number of storage partitions that need to be updated for the split data batch based on the split results; Based on the preset data scheduling time, the number of storage partitions that need to be updated and the data generation time, the complete delayed data of the events of the associated vehicle models are inserted into the updateable target table of the data lake in a serial batch manner to complete the scheduling of the delayed data.

5. The vehicle networking data scheduling method according to claim 4, characterized in that: The data splitting of the complete event delay data of the associated vehicle model includes: The complete event delay data of the associated vehicle model is split according to the generation time of the event data contained in the delay data, and / or the complete event delay data of the associated vehicle model is split according to the duration of the data's arrival in the warehouse.

6. The vehicle networking data scheduling method according to claim 5, characterized in that: Based on the preset data scheduling time, the number of storage partitions to be updated, and the data generation time, the complete delayed event data of the associated vehicle model is inserted into the updateable target table of the data lake in a serial batch manner to complete the scheduling batch of the delayed data, including: If the number of storage partitions that need to be updated for the data batch after the splitting result of the data splitting of the event complete delay data of the associated vehicle model according to the length of the data storage time is greater than or equal to the first preset threshold, the event complete delay data of the associated vehicle model is split according to the generation time of the event data contained in the delay data to determine the first splitting result; If the preset data scheduling time starts at 0 o'clock, the complete event delay data of the associated vehicle model is sorted according to the generation time sequence of the event data, and combined with the first split result, the complete event delay data of the associated vehicle model is inserted into the updateable target table of the data lake in a serial batch manner to complete the scheduling of the delay data; if the preset data scheduling time starts at 1 o'clock, the complete event delay data of the associated vehicle model is inserted into the updateable target table of the data lake in a serial batch manner to complete the scheduling of the delay data; If the number of storage partitions that need to be updated for the split data batch after the split is calculated by performing data splitting on the complete delayed event data of the associated vehicle model according to the length of the data's storage time is less than or equal to a second preset threshold, then after performing data splitting on the delayed data according to the length of the data's storage time, determine a second splitting result; If the preset data scheduling time starts at 0 o'clock, combined with the second splitting result, the event complete delay data of the associated vehicle model is sorted in the order of the generation time of the event data, and the event complete delay data of the associated vehicle model is inserted into the updateable target table of the data lake in a serial batch manner to complete the scheduling and batching of the delay data; if the preset data scheduling time starts at 1 o'clock, the event complete delay data of the associated vehicle model with the data generation date of the day is inserted into the updateable target table of the data lake in a serial batch manner to complete the scheduling and batching of the delay data.

7. The vehicle networking data scheduling method according to claim 1, characterized in that: The method further includes: setting a non-scheduled time interval for batching event data that failed to be batched on other dates and delayed data that falls within the non-scheduled time interval.

8. The vehicle networking data scheduling method according to claim 1, characterized in that: The method also includes: dynamically allocating data processing resources and / or network transmission resources that can be used by the data batch process within the preset data scheduling time.

9. A vehicle networking data scheduling device, characterized in that: include: An extraction module is used to determine delayed data and / or non-delayed data in the vehicle-side networked data to be batched based on the storage time of the data; A determination module, used to determine the time attributes of delayed data and non-delayed data in the vehicle-side networked data to be run, wherein the time attributes at least include corresponding date information and corresponding hour information; An association module, for performing data aggregation and vehicle model association processing on the delayed data and / or non-delayed data respectively according to the time attribute, and determining event complete delayed data of the associated vehicle model and / or event complete non-delayed data of the associated vehicle model; A scheduling module is used to determine target data from the complete delayed data of the event and / or the complete non-delayed data of the event of the associated vehicle model based on the preset data scheduling time and the time attributes of each delayed data and non-delayed data, and insert the target data into an updateable target table of the data lake to complete the scheduling and batching of the vehicle-side networked data to be batched.

10. A computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, and when the computer executable instructions are executed, the vehicle network data scheduling method as described in any one of claims 1-8 is executed.

Images