Method, apparatus and device for processing cloud resources

Abstract

The present application relates to a cloud resource processing method, device and equipment. The cloud resource processing method comprises: determining time characteristics, regional characteristics and vehicle type characteristics based on time information, regional information and vehicle type information of the cloud resource to be processed; processing the time characteristics, regional characteristics and vehicle type characteristics through a prediction model to predict vehicle message flow to be input to a cloud device and obtain a prediction result; the prediction result is used to represent the size of the current vehicle message flow to be input to the cloud device; and the configuration of the cloud resource is adjusted based on the prediction result to match the prediction result. In the case where the current vehicle message flow to be input to the cloud device represented by the prediction result is greater than a second threshold, the consumer instance in the cloud resource is increased; in the case where the current vehicle message flow to be input to the cloud device represented by the prediction result meets a first preset condition, the consumer instance in the cloud resource is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet of Vehicles data synchronization, and particularly relates to a cloud resource processing method, device and equipment. BACKGROUND

[0002] With the rapid development of Internet of Vehicles technology and the popularity of intelligent vehicles, vehicle messages generated by vehicles are growing explosively, reaching hundreds of billions per day. These vehicle messages are collected together to form vehicle message traffic, which is crucial for vehicle state monitoring, driving behavior analysis and intelligent traffic management. SUMMARY

[0003] The present application provides a cloud resource processing method, device and equipment.

[0004] To achieve the above-mentioned purpose, the technical solution adopted by the present application is as follows:

[0005] In a first aspect, the present application provides a cloud resource processing method, which is applied to a cloud device. The method comprises: determining time features, regional features and vehicle type features based on time information, regional information and vehicle type information of the cloud resource to be processed; processing the time features, the regional features and the vehicle type features through a prediction model to predict vehicle message traffic to be input to the cloud device, and obtaining a prediction result; the prediction result is used to represent the size of the current vehicle message traffic to be input to the cloud device; adjusting the configuration of the cloud resource based on the prediction result, so that the configuration of the cloud resource matches the prediction result; wherein the adjusting the configuration of the cloud resource based on the prediction result comprises: increasing consumer instances in the cloud resource in the case that the current vehicle message traffic to be input to the cloud device represented by the prediction result is greater than a second threshold; reducing the consumer instances in the cloud resource in the case that the current vehicle message traffic to be input to the cloud device represented by the prediction result meets a first preset condition; wherein the first preset condition comprises: the current vehicle message traffic to be input to the cloud device represented by the prediction result is less than a third threshold, or the current vehicle message traffic to be input to the cloud device represented by the prediction result is less than the third threshold and exceeds a specified time length; the more the number of consumer instances, the faster the ability of the cloud resource to process the vehicle message traffic; the fewer the number of consumer instances, the slower the ability of the cloud resource to process the vehicle message traffic; wherein the third threshold is less than or equal to the second threshold.

[0006] It can be understood that the scheme provided by the embodiments of the present application is that the vehicle message flow corresponding to different times is different, the vehicle message flow corresponding to different regions is different, and the vehicle message flow corresponding to different vehicle types is also different. By determining the time information, region information and vehicle type information to be processed by the cloud resource, the prediction target of the prediction model can be clearly predicted, that is, what vehicle message flow of what region and what time of what vehicle needs to be predicted by the prediction model. Based on the determination of the time information, the region information and the vehicle type information, the time feature, the region feature and the vehicle type feature are determined. The vehicle message flow to be input to the cloud device is predicted by processing the time feature, the region feature and the vehicle type feature by the prediction model, and a prediction result is obtained. The vehicle message flow is dynamically predicted. The prediction result is used to represent the size of the current vehicle message flow to be input to the cloud device. Based on the prediction result, the configuration of the cloud resource is adjusted to match the prediction result. On the one hand, the fluctuation of the vehicle message flow can be coped with, and the phenomenon that the vehicle message flow cannot be processed due to the surge of the vehicle message flow will not occur. On the other hand, the cloud resource can be cooperated to enable the cloud resource to process the predicted current vehicle message flow in time, improve the synchronization, and avoid the waste of the cloud resource. In addition, in the case that the current vehicle message flow to be input to the cloud device is greater than the second threshold, the consumer instance in the cloud resource is increased, which can ensure that the vehicle message flow can be processed in time, improve the processing efficiency of the vehicle message flow, and reduce the backlog of the vehicle message flow. In the case that the current vehicle message flow to be input to the cloud device is less than the third threshold, it indicates that the current vehicle message flow is relatively small. By reducing the consumer instance in the cloud resource, the redundant consumer instance can be released, and the resource waste can be reduced. Alternatively, considering the misjudgment, in order to reduce the occurrence of misjudgment, the time can be considered, that is, when the predicted current vehicle message flow is less than the third threshold and exceeds the specified time length, the consumer instance in the cloud resource is reduced.

[0007] In some embodiments, the adjusting the configuration of the cloud resource based on the prediction result comprises: in the case that the current vehicle message flow to be input to the cloud device represented by the prediction result exceeds the first processing capability of the cloud resource, increasing a throughput node in the cloud resource; in the case that the current vehicle message flow to be input to the cloud device represented by the prediction result is lower than the second processing capability of the cloud resource, reducing a throughput node in the cloud resource; and the first processing capability is higher than the second processing capability.

[0008] It can be understood that the scheme provided by the embodiments of the present application limits the adjustment according to the prediction result to the throughput nodes in the cloud resource. The throughput nodes of the cloud resource represent the processing capacity of the cloud resource for the predicted current vehicle message flow. In the case that the current vehicle message flow to be input to the cloud device exceeds the first processing capacity of the cloud resource, increasing the throughput nodes can cope with the surge of vehicle message flow and ensure that the vehicle message flow can be completely processed to improve the processing efficiency of the cloud resource. In the case that the current vehicle message flow to be input to the cloud device is lower than the second processing capacity of the cloud resource, reducing the throughput nodes can ensure that the vehicle message flow can be completely processed and release the redundant throughput nodes to reduce the waste of the cloud resource.

[0009] In some embodiments, the adjusting the configuration of the cloud resource based on the prediction result comprises: in the case that the first vehicle message flow in the current vehicle message flow to be input to the cloud device represented by the prediction result exceeds a first threshold, determining an increase amount of the first vehicle message flow; and increasing a partition for a first region in the cloud resource corresponding to processing the first vehicle message flow according to the increase amount of the first vehicle message flow, so that the first region after the increase in partition can process the first vehicle message flow.

[0010] It can be understood that the scheme provided by the embodiments of the present application limits the adjustment according to the prediction result to the first region of the cloud resource. By determining the increase amount of the first vehicle message flow in the case that the first vehicle message flow in the current vehicle message flow to be input to the cloud device represented by the prediction result exceeds a first threshold, and increasing a partition for the first region according to the increase amount of the first vehicle message flow, so that the first region after the increase in partition can process the first vehicle message flow, the surge of the first vehicle message flow can be coped with, the stability of the cloud resource for processing the vehicle message flow is improved, and the occurrence of delay or backlog is reduced.

[0011] In some embodiments, the adjusting the configuration of the cloud resource based on the prediction result comprises: based on the current vehicle message flow to be input to the cloud device represented by the prediction result, determining that the current vehicle message flow belongs to a trough period of vehicle message flow; based on the priority of the vehicle message flow being processed in the cloud resource, determining at least two low-activity regions in the cloud resource; determining a target region and an idle region in the at least two low-activity regions; the target region is used to process the current vehicle message flow and the vehicle message flow being processed in the two low-activity regions; and the idle region is used as a backup region.

[0012] It can be understood that the scheme provided by the embodiments of the present application limits the cloud resource area to be adjusted according to the prediction result. The cloud resource has multiple areas, and each area can be used to process vehicle message traffic. In a case where it is determined that the current vehicle message traffic belongs to a low-traffic period of vehicle message traffic, it is indicated that the current vehicle message traffic is relatively small. The target area and the idle area can be determined in at least two low-activity areas, the target area is used to process the current vehicle message traffic and the vehicle message traffic being processed in the two low-activity areas, and the idle area is used as a backup area to avoid resource occupation. That is, in a case where it is determined that the current vehicle message traffic belongs to a low-traffic period of vehicle message traffic, the vehicle message traffic being processed in at least two low-activity areas is combined and continues to be processed in the target area, so that the remaining low-activity areas can be released. The released low-activity area is called an idle area and can be used as a backup to expand in a case where subsequent vehicle message traffic surges or vehicle message traffic is in a peak period, so as to ensure that the vehicle message traffic can be processed in time and reduce the occurrence of delay or backlog.

[0013] In some embodiments, the adjusting the configuration of the cloud resource based on the prediction result comprises: dividing each vehicle message in the current vehicle message traffic to be input to the cloud device according to the importance of the vehicle message traffic, determining the importance level of each vehicle message; determining the area processing high-importance-level vehicle messages as an independent area in the cloud resource, or determining the throughput node processing the high-importance-level vehicle messages as an independent throughput node; the high-importance-level vehicle message is a vehicle message including fault alarm information; the low-importance-level vehicle message is a vehicle message including regular state information.

[0014] It can be understood that the scheme provided by the embodiments of the present application limits the different resources to be allocated according to the importance of each vehicle message in the vehicle traffic message. By determining the processing area of the high-importance-level vehicle message as an independent area in the cloud resource or determining the throughput node processing the high-importance-level vehicle message as an independent throughput node, it can be ensured that the high-importance-level vehicle message is processed in time and the delay is reduced. By compressing the low-importance-level vehicle message before transmission, resource waste can be reduced.

[0015] In some embodiments, after the configuration of the cloud resource is adjusted based on the prediction result to match the prediction result, the method further comprises: processing, based on the cloud resource, the prediction result to represent the current vehicle message flow to be input to the cloud device to obtain vehicle message flow to be sent; determining target compression algorithms corresponding to vehicle messages of different importance levels based on the importance levels of the vehicle messages in the vehicle message flow to be sent; and transmitting the vehicle messages of different importance levels after compression by the corresponding target compression algorithms; wherein, for vehicle messages of high importance level in the vehicle message flow to be sent, the target compression algorithm is a low-delay compression algorithm; the vehicle messages of high importance level are vehicle messages including fault alarm information; and for vehicle messages of low importance level in the vehicle message flow to be sent, the target compression algorithm is a high-compression-rate algorithm; the vehicle messages of low importance level are vehicle messages including regular state information.

[0016] It can be understood that the scheme provided by the embodiments of the present application limits the vehicle message flow to be sent to be compressed before transmission, which can release bandwidth resources and improve transmission rate. By using a low-delay compression algorithm for vehicle messages of high importance level in the vehicle message flow to be sent, the transmission efficiency and real-time performance can be balanced; and by using a high-compression-rate algorithm for vehicle messages of low importance level in the vehicle message flow to be sent, the transmission amount can be reduced.

[0017] In some embodiments, the transmission of the vehicle messages of different importance levels after compression by the corresponding target compression algorithms comprises: determining health scores of at least two candidate transmission paths of the vehicle message flow to be sent; selecting, from the at least two candidate transmission paths, a transmission path with the highest health score as a target transmission path; and transmitting the vehicle messages of different importance levels after compression by using the target transmission path; wherein, the health of the transmission path is used to evaluate network delay, network bandwidth and network load of the transmission path.

[0018] It can be understood that the scheme provided by the embodiments of the present application limits the transmission, selects a transmission path with the highest health score as a target transmission path for transmission, which can ensure the stability of transmission, improve the transmission rate, reduce the delay, and thus can make the synchronization of transmission faster.

[0019] In some embodiments, the determining the health score of the at least two candidate transmission paths of the vehicle message traffic to be sent comprises: for each of the at least two candidate transmission paths, performing: determining network latency, network bandwidth, and network load of the candidate transmission path; and determining the health score of the candidate transmission path according to the network latency, the network bandwidth, and the network load of the candidate transmission path.

[0020] It can be understood that the scheme provided by the embodiments of the present application limits how to score the health of the transmission path. The network latency of the candidate transmission path can be determined to know the time efficiency of transmitting data through the candidate network. The network bandwidth of the candidate transmission path can be determined to know the transmission capacity of data per unit time of the candidate transmission path. The network load of the candidate transmission path can be determined to know the use pressure of the current network resource of the candidate transmission path. The health score of the transmission path is determined according to the network latency, the network bandwidth, and the network load of the candidate transmission path, which can evaluate the real-time performance, throughput capacity, and stability of the candidate transmission path. Therefore, the transmission path with good real-time performance, strong throughput capacity, and strong stability can be selected from the candidate transmission paths as the target transmission path for transmission, which can improve the transmission rate while reducing the occurrence of congestion.

[0021] In some embodiments, the method further comprises: in the case that the target transmission path is abnormal, switching to a backup transmission path to transmit the compressed vehicle messages of different importance levels through the backup transmission path; and the backup target transmission path is the candidate transmission path with the second highest health score.

[0022] It can be understood that the scheme provided by the embodiments of the present application can further improve the stability of transmission by switching to a backup transmission path for transmission in the case that the target transmission path is abnormal.

[0023] In a second aspect, the embodiments of the present application provide a cloud resource processing apparatus. The apparatus is deployed in a cloud device. The apparatus comprises a first determining unit and a first adjusting unit. The first determining unit is configured to determine time features, regional features, and vehicle type features based on time information, regional information, and vehicle type information of a cloud resource to be processed. The first adjusting unit is configured to process the time features, the regional features, and the vehicle type features through a prediction model, predict vehicle message traffic to be input to the cloud device, and obtain a prediction result. The configuration of the cloud resource is adjusted based on the prediction result, so that the configuration of the cloud resource matches the prediction result. The prediction result is used to represent the size of the current vehicle message traffic to be input to the cloud device.

[0024] In a third aspect, an electronic device is provided, and the electronic device includes a memory and a processor. The memory stores a computer program or instructions. When the computer program or the instructions are executed by the processor, the method of the first aspect is implemented.

[0025] In a fourth aspect, a computer readable storage medium is provided, and the storage medium stores a computer program that can be run on a processor. When the processor executes the program, the method of the first aspect is implemented.

[0026] In a fifth aspect, a computer program product is provided, and the computer program product includes a computer program or instructions. When the computer program or the instructions are executed, the method of the first aspect is implemented. BRIEF DESCRIPTION OF DRAWINGS

[0027] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application. It is apparent that the accompanying drawings are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort based on the accompanying drawings.

[0028] The flowchart shown in the accompanying drawings is only an exemplary illustration, and is not necessarily required to include all contents and operations / steps, nor is it necessarily required to be executed in the order described. For example, some operations / steps can be further divided, and some operations / steps can be combined or partially combined, so that the actual execution order can be changed according to the actual situation.

[0029] Figure 1 A flowchart of a cloud resource processing method provided by an embodiment of the present application is shown in the figure;

[0030] Figure 2 A flowchart of a prediction model training method provided by an embodiment of the present application is shown in the figure;

[0031] Figure 3 A flowchart of a cloud resource dynamic adjustment method based on a prediction model provided by an embodiment of the present application is shown in the figure;

[0032] Figure 4 A structural diagram of a cloud resource processing device provided by an embodiment of the present application is shown in the figure;

[0033] Figure 5 A structural diagram of a prediction model training device provided by an embodiment of the present application is shown in the figure;

[0034] Figure 6 A structural diagram of an electronic device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0035] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will further describe the specific technical solutions of the present application with reference to the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0037] In the following description, “some embodiments”, “the embodiment”, “the embodiments of the present application” and the like are described, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0038] The “first, second, third” and the like appearing in the embodiments of the present application do not have a specific meaning (such as no order, nor represent a special limitation on the number of devices in the embodiments of the present application), but only for the convenience of clearly describing the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0039] Before the embodiments of the present application are further described in detail, the terms and terms that can be involved in the embodiments of the present application are described, and the terms and terms involved in the embodiments of the present application are applicable to the following explanations.

[0040] Distributed Event Streaming Platform: The core role is to process, store and distribute large-scale event data in real time, providing efficient and reliable data pipelines for modern distributed systems. Commonly used distributed event streaming platforms include Kafka.

[0041] Kafka: Through decoupling, asynchronous communication and real-time data stream processing, it solves the data flow and collaboration problems in modern distributed systems.

[0042] Cloud resources: Cloud resources include public clouds and / or private clouds, wherein the public cloud is generally deployed on the cloud; the private cloud can be deployed on the cloud or off the cloud, and the latter will be exemplarily described.

[0043] For the convenience of understanding the technical solutions of the embodiments of the present application, the related technologies or terms of the embodiments of the present application are described below. The following related technologies or related terms can be combined with the technical solutions of the embodiments of the present application as optional solutions, and all of them belong to the protection scope of the embodiments of the present application.

[0044] With the rapid development of Internet of Vehicles technology and the popularity of intelligent vehicles, the vehicle messages generated by vehicles show explosive growth, reaching hundreds of billions per day. These vehicle messages are collected together to form vehicle message traffic, and the vehicle message traffic is crucial for vehicle state monitoring, driving behavior analysis, intelligent traffic management and other applications.

[0045] Based on this, the inventors of the present application found through research and analysis that the related solutions cannot dynamically adjust the configuration of cloud resources based on predicted vehicle message traffic.

[0046] Based on this, the embodiments of the present application provide the following cloud resource processing method, device and equipment, etc.

[0047] Embodiments of the cloud resource processing method:

[0048] It should be noted that the cloud resource processing method provided by the embodiments of the present application is applied to a cloud device.

[0049] Figure 1 A flowchart of a cloud resource processing method provided by the embodiments of the present application is shown in FIG. 1, which includes the following steps: Figure 1

[0050] S101, determining time features, regional features and vehicle type features based on time information, regional information and vehicle type information to be processed by the cloud resources;

[0051] S102, processing the time features, regional features and vehicle type features by a prediction model to predict vehicle message traffic to be input to the cloud device, and obtaining a prediction result; the prediction result is used to represent the size of the current vehicle message traffic to be input to the cloud device;

[0052] S103, adjusting the configuration of the cloud resources based on the prediction result, so that the configuration of the cloud resources matches the prediction result.

[0053] ​It can be understood that the vehicle message flow corresponding to different times is different, the vehicle message flow corresponding to different regions is different, and the vehicle message flow corresponding to different vehicle types is also different. By determining the time information, the region information, and the vehicle type information to be processed by the cloud resource, the prediction target of the prediction model can be determined, that is, what vehicle message flow of what region at what time of what vehicle needs to be predicted by the prediction model. Based on the determination of the time information, the region information, and the vehicle type information, the time feature, the region feature, and the vehicle type feature are determined. The prediction model processes the time feature, the region feature, and the vehicle type feature to predict the vehicle message flow to be input to the cloud device, obtains a prediction result, and dynamically predicts the vehicle message flow. The prediction result is used to represent the size of the current vehicle message flow to be input to the cloud device. Based on the prediction result, the configuration of the cloud resource is adjusted to match the prediction result, on the one hand, the fluctuation of the vehicle message flow can be coped with, and the phenomenon that the vehicle message flow cannot be processed due to the surge of the vehicle message flow can be avoided; on the other hand, the cloud resource can be cooperated to enable the cloud resource to process the predicted current vehicle message flow in time, improve the synchronization, and avoid the waste of the cloud resource.

[0054] The further optional embodiments of each of the above steps and related terms are described below.

[0055] S101, based on the time information, the region information, and the vehicle type information to be processed by the cloud resource, determining the time feature, the region feature, and the vehicle type feature.

[0056] Exemplarily, the cloud resource includes a Kafka cluster on the cloud and / or a Kafka cluster under the cloud. Subsequently, the cloud resource includes the Kafka cluster on the cloud and the Kafka cluster under the cloud is taken as an example for exemplary description.

[0057] It can be understood that, since the vehicle message flow corresponding to different times is different, the vehicle message flow corresponding to different regions is different, and the vehicle message flow corresponding to different vehicle types is also different, by determining the time information, the region information, and the vehicle type information to be processed by the cloud resource, the prediction target of the prediction model can be determined, that is, what vehicle message flow of what region at what time of what vehicle needs to be predicted by the prediction model.

[0058] In an optional embodiment, the processing based on the time information, the region information, and the vehicle type information can include that the prediction model determines the time feature from the time information, determines the region feature from the region information, and determines the vehicle type feature from the vehicle type information.

[0059] Exemplarily, the time feature can include but is not limited to a timestamp, an hour, a day of the week, and whether it is a holiday, etc. The timestamp is used to reflect the generation time of the vehicle message data.

[0060] Exemplarily, the regional features can include, but are not limited to, region numbers, region types, and vehicle message flow proportions of regions, etc.; wherein the vehicle message flow proportion of a region refers to a ratio between vehicle message flow of the region in a time period and vehicle message flow of the whole country in the time period.

[0061] Exemplarily, the vehicle type features can include, but are not limited to, vehicle type numbers, vehicle states, and frequencies of generation of single vehicle message data, etc.; wherein the vehicle types can include, but are not limited to, passenger cars, commercial vehicles, new energy vehicles, etc.; wherein the vehicle states can include, but are not limited to, driving, idling, and engine off, etc.

[0062] Since the time features, the regional features, and the vehicle type features are used to train the model in the process of obtaining the prediction model, after the time information, the regional information, and the vehicle type information are determined, the prediction model still needs to perform feature extraction processing on the time information, the regional information, and the vehicle type information, so as to enable the prediction model to predict the vehicle message flow to be input to the cloud device.

[0063] S102, processing the time features, the regional features, and the vehicle type features by the prediction model to predict the vehicle message flow to be input to the cloud device, and obtaining a prediction result.

[0064] The prediction result is used to represent a size of the current vehicle message flow to be input to the cloud device.

[0065] In an optional embodiment, the processing of the time features, the regional features, and the vehicle type features by the prediction model can be some processing inside the prediction model, that is, necessary processing for obtaining the prediction result, which is related to the structure of the prediction model, for example, convolution processing, etc.

[0066] S103, adjusting the configuration of the cloud resource based on the prediction result, so as to match the configuration of the cloud resource with the prediction result.

[0067] In an optional embodiment, adjusting the configuration of the cloud resource based on the prediction result, so as to match the configuration of the cloud resource with the prediction result can be expanding the cloud resource when the prediction result is large, and shrinking the cloud resource when the prediction result is small.

[0068] The configuration of the cloud resource can be throughput node numbers, partition numbers, and consumer instances, etc., which are not particularly limited in the present application. That is, the throughput node numbers, the partition numbers, and the consumer instances, etc. of the cloud resource can be dynamically adjusted according to the prediction result, so as to cope with the fluctuation of the vehicle message flow and avoid resource waste.

[0069] In an alternative embodiment, adjusting the configuration of the cloud resource based on the prediction result can comprise: in a case where the current vehicle message flow to be input to the cloud device as characterized by the prediction result exceeds a first processing capacity of the cloud resource, increasing throughput nodes (Brokers) in the cloud resource; in a case where the current vehicle message flow to be input to the cloud device as characterized by the prediction result is lower than a second processing capacity of the cloud resource, reducing the throughput nodes in the cloud resource.

[0070] The first processing capacity is higher than the second processing capacity.

[0071] It can be understood that the vehicle message flow is processed by the throughput nodes in the cloud resource, and the vehicle message flow includes a plurality of vehicle message data (i.e. a plurality of vehicle messages), and the amount of vehicle message data that can be processed by one throughput node is limited. Therefore, the number of throughput nodes can be dynamically adjusted according to the predicted current vehicle message flow, so that the predicted current vehicle message flow can be processed in time and not backlog, and resource waste can be reduced.

[0072] For example, the first processing capacity can refer to that the throughput node can process 80% of the preset vehicle message data amount, and the second processing capacity can refer to that the throughput node can process 30% of the preset vehicle message data amount. It can be understood that 80% and 30% are examples of the first processing capacity and the second processing capacity, and are not limited to the first processing capacity and the second processing capacity. The first processing capacity and the second processing capacity can be adjusted according to actual conditions, as long as the first processing capacity is higher than the second processing capacity.

[0073] Further, in an alternative embodiment, the number of Brokers can be dynamically adjusted by calling a cloud platform API.

[0074] For example, the blue-green deployment strategy can be used to ensure that the service is not interrupted during the expansion process.

[0075] It can be understood that the throughput nodes of the cloud resource represent the processing capacity of the cloud resource for the predicted current vehicle message flow. In a case where the current vehicle message flow to be input to the cloud device exceeds the first processing capacity of the cloud resource, increasing the throughput nodes can cope with the surge of the vehicle message flow and ensure that the vehicle message flow can be completely processed, thereby improving the processing efficiency of the cloud resource. In a case where the current vehicle message flow to be input to the cloud device is lower than the second processing capacity of the cloud resource, reducing the throughput nodes can ensure that the vehicle message flow can be completely processed, thereby releasing the redundant throughput nodes and reducing the waste of the cloud resource.

[0076] It can be understood that, as a cloud resource, the cloud resource includes the on-cloud Kafka cluster and the off-cloud Kafka cluster, in order to ensure synchronization, the throughput nodes of the on-cloud Kafka cluster are increased, and correspondingly, the throughput nodes of the off-cloud Kafka cluster should also be increased, so as to ensure that the vehicle message flow processed on the cloud can be timely received by the off-cloud, and the occurrence of missed connection is reduced. The throughput nodes of the on-cloud Kafka cluster are reduced, and correspondingly, the throughput nodes of the off-cloud Kafka cluster should also be reduced, so as to ensure that the resources on the cloud and off the cloud are not wasted.

[0077] In an optional embodiment, adjusting the configuration of the cloud resource based on the prediction result comprises: in the case that the prediction result represents that the first vehicle message flow in the current vehicle message flow to be input to the cloud device exceeds the first threshold, determining an increase amount of the first vehicle message flow; and increasing partitions for a first region in the cloud resource corresponding to processing the first vehicle message flow according to the increase amount of the first vehicle message flow, so that the first region after the increase of the partitions can process the first vehicle message flow.

[0078] In the embodiment, the first region is any one region in the cloud resource.

[0079] Further, as an optional embodiment, the increase amount of the first vehicle message flow can be a difference value obtained by subtracting the first threshold from the first vehicle message flow; as another optional embodiment, the increase amount of the first vehicle message flow is a ratio value obtained by dividing the difference value obtained by subtracting the first threshold from the first vehicle message flow by the first vehicle message flow. The present application does not particularly limit this, and the following will be exemplarily described taking the ratio value obtained by dividing the difference value obtained by subtracting the first threshold from the first vehicle message flow by the first vehicle message flow as an example.

[0080] Further, as an optional embodiment, increasing partitions for the first region according to the increase amount of the first vehicle message flow can be allocating the newly added partitions in proportion according to the increase amount of the first vehicle message flow. Exemplarily, if the predicted first vehicle message flow is increased by 20%, then 20% of the partitions are added to the first region. It can be understood that 20% is only an example of the increase amount of the first vehicle message flow, and is not a limitation on the increase amount of the first vehicle message flow. In actual application, the increase amount of the first vehicle message flow is calculated according to actual conditions.

[0081] Exemplarily, the newly added partitions can be realized by calling a Kafka administrator (Admin) application programming interface (Application Programming Interface, API).

[0082] By using programmed means rather than manual commands to manage Kafka, an automated cloud resource coordination mechanism can be achieved.

[0083] It can be understood that, since the vehicle message data volume is continuously generated, if the consumer in the cloud resource cannot timely consume (i.e., process) the generated vehicle message data volume, information accumulation will occur. Therefore, as another optional embodiment, it can be determined that the first vehicle message flow exceeds the first threshold value in a case where the message accumulation volume of the first region exceeds the first preset accumulation volume.

[0084] It can be understood that, by determining the increase of the first vehicle message flow in a case where the first vehicle message flow in the current vehicle message flow to be input to the cloud device represented by the prediction result exceeds the first threshold value, and adding partitions to the first region according to the increase of the first vehicle message flow, the first region after adding the partitions can process the first vehicle message flow, which can cope with the surge of the first vehicle message flow, improve the stability of the cloud resource for processing the vehicle message flow, and reduce the occurrence of delay or backlog.

[0085] It can be understood that, since it is a cloud resource, the cloud resource includes an on-cloud Kafka cluster and an off-cloud Kafka cluster, in order to ensure synchronization, the first region of the on-cloud Kafka cluster adds partitions, correspondingly, the first region of the off-cloud Kafka cluster should also add the same number of partitions, so as to ensure that the vehicle message flow processed by the on-cloud can be timely received by the off-cloud. The first region of the off-cloud Kafka cluster refers to the region corresponding to the first region of the on-cloud Kafka cluster and used to receive the output of the first region of the on-cloud Kafka cluster.

[0086] In an optional embodiment, adjusting the configuration of the cloud resource based on the prediction result comprises: based on the current vehicle message flow to be input to the cloud device represented by the prediction result, determining that the current vehicle message flow belongs to a low period of the vehicle message flow: based on the priority of the vehicle message flow being processed in the cloud resource, determining at least two low-activity regions in the cloud resource; determining a target region and an idle region in the at least two low-activity regions; the target region is used to process the current vehicle message flow and the vehicle message flow being processed in the two low-activity regions; the idle region is used as a backup region.

[0087] The low-activity region is used to process the vehicle message flow with low priority. It can be understood that the vehicle message flow with low priority is composed of vehicle messages with low priority.

[0088] The priority of the vehicle message is related to the importance of the vehicle message: the higher the importance of the vehicle message, the higher the priority of the vehicle message; the lower the importance of the vehicle message, the lower the priority of the vehicle message.

[0089] As an optional embodiment, the vehicle messages can be divided into high-importance-level vehicle messages and low-importance-level vehicle messages according to the importance of the services. The high-importance-level vehicle messages can be vehicle messages including fault alarm information, and the low-importance-level vehicle messages can be vehicle messages including routine information.

[0090] The idle area can be used as a backup area to process vehicle message traffic exceeding the processing capacity of the target area, or to merge with the first area to expand the capacity when it is necessary to increase the partition of the first area.

[0091] It can be understood that there are multiple areas of cloud resources, each of which can be used to process vehicle message traffic. In a case where it is determined that the current vehicle message traffic belongs to a trough period of vehicle message traffic, it means that the current vehicle message traffic is relatively small. By determining a target area and an idle area in at least two low-activity areas, the target area is used to process the current vehicle message traffic and the vehicle message traffic being processed in the two low-activity areas, and the idle area is used as a backup area to avoid resource occupation. In other words, in a case where it is determined that the current vehicle message traffic belongs to a trough period of vehicle message traffic, the vehicle message traffic being processed in at least two low-activity areas is merged and continues to be processed in the target area, thereby releasing the remaining low-activity areas, which are referred to as idle areas and can be used as backups to expand the capacity in a case where subsequent vehicle message traffic surges or vehicle message traffic is in a peak period, so as to ensure that vehicle message traffic can be processed in time and reduce the occurrence of delay or backlog.

[0092] It can be understood that, since it is a cloud resource, the cloud resource includes an on-cloud Kafka cluster and an off-cloud Kafka cluster. Correspondingly, the low-activity areas of the on-cloud Kafka cluster are merged, and the low-activity areas of the off-cloud Kafka cluster also need to be merged to save resources on the cloud and off the cloud.

[0093] In an optional embodiment, adjusting the configuration of the cloud resource based on the prediction result includes: in a case where the current vehicle message traffic to be input to the cloud device represented by the prediction result is greater than a second threshold, increasing the consumer instances in the cloud resource; and in a case where the current vehicle message traffic to be input to the cloud device represented by the prediction result satisfies a first preset condition, reducing the consumer instances in the cloud resource.

[0094] It can be understood that the more the number of consumer instances, the faster the cloud resource processes the vehicle message flow; the less the number of consumer instances, the slower the cloud resource processes the vehicle message flow.

[0095] As an optional embodiment, the first preset condition can include that the current vehicle message flow to be input to the cloud device characterized by the prediction result is less than a third threshold.

[0096] The third threshold is less than or equal to the second threshold.

[0097] In order to reduce the occurrence of misjudgment, as another optional embodiment, the first preset condition can include that the current vehicle message flow to be input to the cloud device characterized by the prediction result is less than a third threshold and exceeds a specified duration.

[0098] Exemplarily, the specified duration can be 10 minutes. It can be understood that 10 minutes is only an example of the specified duration, and the specified duration is not limited to 10 minutes. In actual application, the specified duration can be set according to actual needs.

[0099] Further, as an optional embodiment, the current vehicle message flow to be input to the cloud device can be monitored by a Kafka consumer group API, and the number of consumer instances can be adjusted in combination with a container orchestration tool.

[0100] It can be understood that in the cloud resource, the role of the consumer instance is to actively request data from the throughput node and process it. The more the consumer instances, the higher the throughput capacity of the throughput node.

[0101] It can be understood that since the vehicle message data volume is continuously generated, if the consumer in the cloud resource cannot timely consume (i.e., process) these generated vehicle message data volume, information accumulation will occur. Therefore, as another optional embodiment, the number of consumer instances can be automatically increased or decreased based on the message accumulation amount.

[0102] Exemplarily, when the message accumulation amount (Lag) exceeds a second preset accumulation amount, it is determined that the current vehicle message flow to be input to the cloud device is greater than the second threshold, at which time the number of consumer instances can be increased; when the message accumulation amount is less than the second preset accumulation amount, it is determined that the current vehicle message flow to be input to the cloud device is less than the third threshold, at which time the number of consumer instances can be reduced, or when the message accumulation amount is less than the second preset accumulation amount and exceeds 10 minutes, it is determined that the current vehicle message flow to be input to the cloud device is less than the third threshold and exceeds 10 minutes, at which time the number of consumer instances can also be reduced.

[0103] Exemplarily, the message accumulation amount can also be monitored through the Kafka Consumer Group API, and the number of consumer instances is adjusted in combination with the container orchestration tool.

[0104] It can be understood that, in the case that the current vehicle message flow to be input to the cloud device is greater than the second threshold, increasing the consumer instances in the cloud resources can ensure that the vehicle message flow can be processed in time, improve the processing efficiency of the vehicle message flow, and reduce the backlog of the vehicle message flow; in the case that the current vehicle message flow to be input to the cloud device is less than the third threshold, it indicates that the current vehicle message flow is relatively small, and by reducing the consumer instances in the cloud resources, the redundant consumer instances can be released, and resource waste can be reduced; or, considering misjudgment, in order to reduce the occurrence of misjudgment, the time can be considered, that is, when the predicted current vehicle message flow is less than the third threshold and exceeds the specified time length, the consumer instances in the cloud resources are reduced.

[0105] It can be understood that, since it is cloud resources, the cloud resources include the on-cloud Kafka cluster and the off-cloud Kafka cluster, and correspondingly, the consumer instances of the off-cloud Kafka cluster also need to be reduced, so that the resources of the on-cloud Kafka cluster and the resources of the off-cloud Kafka cluster are both saved.

[0106] In an optional embodiment, adjusting the configuration of the cloud resources based on the prediction result comprises: triggering partition reassignment according to the increase or decrease of the consumer instances, to ensure load balancing of each consumer. For example, when the consumer instances are increased, the partitions are also increased for the area where the consumer instances are located; for another example, when the consumer instances are reduced, the partitions are also reduced for the area where the consumer instances are located.

[0107] Exemplarily, the sticky assignment strategy of the Kafka StickyAssignor can be used to keep the assignment relationship through stickiness, greatly optimizing the process of consumer rebalance, and reducing unnecessary partition movement and related overheads.

[0108] In an optional embodiment, adjusting the configuration of the cloud resources based on the prediction result comprises: dividing each vehicle message in the current vehicle message flow to be input to the cloud device represented by the prediction result according to the importance of the vehicle message service, to determine the importance level of each vehicle message; determining the area for processing the vehicle messages of high importance level as an independent area in the cloud resources, or determining the throughput node for processing the vehicle messages of high importance level as an independent throughput node; and compressing the vehicle messages of low importance level before transmission.

[0109] The vehicle message with the high importance level is a vehicle message including fault alarm information, and the vehicle message with the low importance level is a vehicle message including normal state information.

[0110] It can be understood that, by determining the processing area of the vehicle message with the high importance level as an independent area in the cloud resource or determining the throughput node processing the vehicle message with the high importance level as an independent throughput node, it can be ensured that the vehicle message with the high importance level is processed in time and the delay is reduced; by compressing the vehicle message with the low importance level before transmission, resource waste can be reduced.

[0111] It can be understood that, since it is a cloud resource, the cloud resource includes an on-cloud Kafka cluster and an off-cloud Kafka cluster, and the independent throughput node or the independent area is allocated to the vehicle message with the high importance level in the on-cloud Kafka cluster, and correspondingly, the independent throughput node or the independent area is also allocated to the vehicle message with the high importance level in the off-cloud Kafka cluster; the vehicle message with the low importance level in the on-cloud Kafka cluster is compressed before transmission, and correspondingly, the vehicle message with the low importance level in the off-cloud Kafka cluster also needs to be compressed before transmission, so as to ensure the synchronization of on-cloud and off-cloud transmission and the consistency of processing capacity.

[0112] It can be understood that, after adjusting the cloud resource configuration based on the prediction result, data transmission is still needed, in order to improve the transmission efficiency and stability, as an optional embodiment, the cloud resource processing method provided by the embodiment of the application further includes a transmission method.

[0113] The transmission method dynamically selects the optimal data transmission path by comprehensively analyzing the network state, data flow characteristics and resource load situation. First, the network delay, bandwidth utilization and node load indicators between the on-cloud and off-cloud Kafka clusters can be collected in real time to build a multi-dimensional evaluation model. Second, based on the evaluation result, the transmission path is dynamically sorted, and the path with low delay and high availability is preferentially selected. For example, when it is detected that the network delay of a certain path exceeds the threshold, the backup path is automatically switched to. In addition, the transmission method also supports a differentiated scheduling strategy, which allocates exclusive network channels to high-priority data to ensure the real-time synchronization of critical messages.

[0114] As an optional embodiment, the transmission method can include the following steps:

[0115] S103A1, processing the current vehicle message flow to be input to the cloud device based on the prediction result, to obtain a vehicle message flow to be sent.

[0116] S103A2, determining a target compression algorithm corresponding to vehicle messages with different importance levels based on the importance levels of the vehicle messages in the vehicle message flow to be sent.

[0117] In an optional embodiment, for the high-importance vehicle messages in the vehicle message traffic to be sent, the target compression algorithm of the high-importance vehicle messages is determined to be a low-delay compression algorithm; for the low-importance vehicle messages in the vehicle message traffic to be sent, the target compression algorithm of the low-importance vehicle messages is determined to be a high-compression-rate algorithm.

[0118] As stated previously, the high-importance vehicle messages are fault alarm messages, and the low-importance vehicle messages are regular messages. Exemplarily, the low-delay compression algorithm can be a Snappy compression, and the high-compression-rate algorithm can be a Z standard compression algorithm.

[0119] It can be understood that by using the low-delay compression algorithm for the high-importance vehicle messages in the vehicle message traffic to be sent, the transmission efficiency and real-time performance can be balanced; by using the high-compression-rate algorithm for the low-importance vehicle messages in the vehicle message traffic to be sent, the transmission amount can be reduced.

[0120] S103A3, the vehicle messages of different importance levels are transmitted after being compressed by the corresponding target compression algorithms.

[0121] It can be understood that when transmission is performed, there are multiple transmission paths, and since the transmission has not been performed, the transmission paths can be referred to as candidate transmission paths. As an optional embodiment, the health degree of each candidate path can be scored, and then the candidate transmission path with the highest health degree score is selected as the target transmission path for transmission; as another optional embodiment, the health degree of at least two candidate paths can be scored, and then the candidate transmission path with the highest health degree score is selected as the target transmission path for transmission.

[0122] The present application does not make special limitations thereon, and in actual application, the selection can be made according to the needs and the occupation of processing resources, i.e., if the processing resources are free and / or it is desired to ensure the optimal transmission efficiency and the most stable transmission, the health degree of each candidate path can be scored, and then the candidate transmission path with the highest health degree score is selected as the target transmission path for transmission; if the processing resources are limited and / or it is desired to have a relatively fast transmission rate and relatively stable transmission, the health degree of at least two candidate paths can be scored, and then the candidate transmission path with the highest health degree score is selected as the target transmission path for transmission.

[0123] For example, the health scores of at least two candidate paths are determined, and then the candidate transmission path with the highest health score is selected as the target transmission path for transmission. It can be understood that, after knowing the principle of determining the health scores of at least two candidate paths and then selecting the candidate transmission path with the highest health score as the target transmission path for transmission, one skilled in the art can easily infer the scheme of determining the health scores of each candidate path and then selecting the candidate transmission path with the highest health score as the target transmission path for transmission, which will not be described here.

[0124] For example, the vehicle messages of different importance levels are compressed by corresponding target compression algorithms and then transmitted, including the following steps:

[0125] S103A3a, determining the health scores of at least two candidate transmission paths of the vehicle message flow to be sent.

[0126] The health of the transmission path is used to evaluate the network delay, network bandwidth and network load of the transmission path.

[0127] The network delay reflects the time efficiency of the candidate transmission path in transmitting data; the network bandwidth reflects the transmission capacity of the candidate transmission path in unit time; and the network load reflects the use pressure of the current network resources of the candidate transmission path.

[0128] Further, as an optional embodiment, for each of the at least two candidate transmission paths, the network delay, network bandwidth and network load of the candidate transmission path are determined, and the health score of the candidate transmission path is determined according to the network delay, network bandwidth and network load of the candidate transmission path.

[0129] For example, the health score of the candidate transmission path can be determined by the following formula (1):

[0130] (1)

[0131] wherein S i represents the health score of the i th candidate transmission path; Normalized( ) represents normalization processing, which scales each index to the range of [0, 1]; RTT i represents the network delay of the i th candidate transmission path; Bandwidth i represents the network bandwidth of the i th candidate transmission path; Load i represents the network load of the i th candidate transmission path; represents the network delay weight; represents the network bandwidth weight; represent the network load weights.

[0132] Further, as an optional embodiment, , and may be adjusted according to actual conditions.

[0133] In an optional embodiment, the size can be dynamically adjusted according to the prediction result. For example, if the predicted current vehicle message flow to be input to the cloud device is at a peak, the (e.g., from 0.4 to 0.5) can be increased, and the can be reduced.

[0134] In another optional embodiment, the can be increased when the network is congested.

[0135] Further, as another optional embodiment, reinforcement learning can be used to optimize the weights , and , and the historical average delay can be minimized.

[0136] As an optional embodiment, the health score of the target path and the remaining candidate paths can be updated once every preset time length to record the actual transmission delay and the prediction deviation. For example, the preset time length can be 5 minutes. It can be understood that 5 minutes is only an example of the preset time length and is not a limitation of the preset time length. In actual application, the preset time length can be set according to actual conditions.

[0137] S103A3b, among the at least two candidate transmission paths, the transmission path with the highest health score is selected as the target transmission path.

[0138] S103A3C, the compressed vehicle messages of different importance levels are transmitted by using the target transmission path.

[0139] It can be understood that the vehicle message flow to be sent is compressed before transmission, which can release bandwidth resources and improve transmission speed.

[0140] It can be understood that the transmission path with the highest health score is selected as the target transmission path for transmission, which can ensure the stability of transmission, improve the transmission speed, reduce the delay, and thus make the transmission synchronization faster.

[0141] It can be understood that the time efficiency of transmitting data through the candidate network can be known by determining the network delay of the candidate transmission path; the transmission capacity of data per unit time of the candidate transmission path can be known by determining the network bandwidth of the candidate transmission path; the current network resource usage pressure of the candidate transmission path can be known by determining the network load of the candidate transmission path; the real-time performance, throughput capacity and stability of the candidate transmission path can be evaluated according to the health score of the candidate transmission path determined by the network delay, network bandwidth and network load of the candidate transmission path, so that the transmission path with good real-time performance, strong throughput capacity and strong stability can be selected from the candidate transmission paths as the target transmission path for transmission, which can improve the transmission rate while reducing the occurrence of congestion.

[0142] In order to improve the stability of transmission, as an optional embodiment, the backup transmission path can also be switched to in the case that the target transmission path is abnormal, so as to transmit the compressed vehicle messages of different importance levels through the backup transmission path.

[0143] Among them, the backup target transmission path is the candidate transmission path with the second highest health score.

[0144] In an example, when the network delay of the target transmission path is greater than the preset network delay, it can be considered that the target transmission path is abnormal; in another example, when the packet loss rate of the target transmission path is greater than the preset packet loss rate, it can be considered that the target transmission path is abnormal.

[0145] Exemplarily, the preset network delay can be 50 milliseconds (ms); and the preset packet loss rate can be 5%. It can be understood that 50 ms is an example of the preset network delay, and is not a limitation on the preset network delay. The setting of the preset network delay can be set according to actual conditions and requirements, and the present application does not make special limitation thereon; 5% is an example of the preset packet loss rate, and is not a limitation on the preset packet loss rate. The setting of the preset packet loss rate can be set according to actual conditions and requirements, and the present application does not make special limitation thereon.

[0146] As another optional embodiment, the transmission method provided by the embodiments of the present application can also realize batch processing. Exemplarily, the prediction result is used to process the current vehicle message flow to be input to the cloud device based on the cloud resource, to obtain the vehicle message flow to be sent. When the vehicle message flow to be sent is less than the preset vehicle message flow, the vehicle message flow to be sent can be combined as a batch message before transmission. Among them, the batch processing window size can be dynamically adjusted according to the prediction result: the window is reduced to reduce the delay in the flow peak period, and the window is expanded to increase the throughput in the trough period.

[0147] As another optional embodiment, the transmission method provided by the embodiment of the application further includes fault recovery and consistency guarantee.

[0148] For example, a multi-level fault recovery mechanism is designed to deal with network interruption or cluster anomalies. When a synchronization failure is detected, the system first attempts automatic retry, and gradually extends the retry interval to avoid the snowball effect. If the retry fails, a local cache mechanism is triggered to temporarily store data to a disk queue, and then re-synchronize after the network is restored.

[0149] The retry failure refers to that the retry is not successful before the number of retries reaches or exceeds the preset number of times.

[0150] To ensure data consistency, the Exactly-Once semantics is realized based on the Kafka transaction mechanism, and the periodic offset check is used to prevent data loss or duplication. In addition, the system records the synchronization state checkpoint, supports fast recovery from the breakpoint, and reduces the resource consumption caused by full synchronization.

[0151] As another optional embodiment, the transmission method provided by the embodiment of the application further includes load balancing technology.

[0152] For example, by dynamically allocating synchronization tasks, the overload of a single node is avoided. A distributed load balancer is deployed between the on-cloud Kafka cluster and the off-cloud Kafka cluster, and the message accumulation amount and processing delay of each throughput node are monitored in real time. Based on the monitoring data, the weighted round robin algorithm is used to allocate the synchronization tasks to the nodes with lighter load, and the flow limiting protection is implemented for the high-load nodes. For the problem of uneven load at the partition level, the message distribution of each partition is analyzed periodically, and the mapping relationship between the partition and the throughput node is automatically adjusted by using the partition reassignment tool provided by Kafka, so that the resource utilization is maximized.

[0153] Through the transmission method and the load balancing technology, the data transmission efficiency and stability between the cross-cloud Kafka clusters are improved. Based on the traffic prediction result and the real-time resource state, the data transmission path, the compression strategy and the fault recovery mechanism can be dynamically optimized, so that the efficient synchronization of vehicle message data between the on-cloud and off-cloud environments is ensured.

[0154] Embodiments of the training method of the prediction model:

[0155] The embodiment of the application further provides a training method of a prediction model, which is the prediction model in the embodiment of the cloud resource processing method.

[0156] Figure 2 The flowchart of the training method of the prediction model provided by the embodiment of the application is shown in Figure 2As shown, the training method of the prediction model includes the following steps:

[0157] S201, based on the time information, regional information and vehicle type information of the historical vehicle message flow, determine the time feature, regional feature and vehicle type feature, form a first feature data set.

[0158] The vehicle message flow has significant time correlation, regional distribution characteristics and vehicle type difference. Different vehicle types correspond to different vehicle message flows, different regions correspond to different vehicle message flows, and different times correspond to different vehicle message flows. Based on the time information, regional information and vehicle type information of the historical vehicle message flow, the time feature, regional feature and vehicle type feature are determined (also can be called extracted) through feature engineering, which can provide data support for subsequent training prediction model.

[0159] Exemplarily, the following steps can be included:

[0160] S201A, collect historical vehicle messages to form historical vehicle message flow.

[0161] It can be understood that the historical vehicle message flow includes a large amount of historical vehicle messages (also known as historical vehicle message data), and the historical vehicle message data can come from the cloud Kafka cluster and the cloud Kafka cluster.

[0162] As an optional embodiment, the collection can be carried out in the following way:

[0163] First, Kafka consumer group subscription: deploy consumer group in cloud Kafka cluster and cloud Kafka cluster, subscribe to vehicle message data in real time, and persist vehicle message data to distributed file system (Hadoop Distributed File System, HDFS);

[0164] Second, batch pulling at regular intervals: in order to avoid the influence of real-time collection on system performance, the batch pulling strategy at regular intervals is adopted, for example, pulling data once every 5 minutes.

[0165] S201B, feature extraction is performed on the historical vehicle flow.

[0166] After obtaining the historical vehicle message flow, the historical vehicle message flow includes time information, regional information and vehicle type information. In an optional embodiment, the historical vehicle message flow can be directly extracted for features. In order to improve the quality of the historical vehicle message flow and reduce the consumption of processing resources, as another optional embodiment, the historical vehicle message flow can be cleaned first and then extracted for features. The present application does not particularly limit this, and the subsequent example is taken as an example of cleaning the historical vehicle message flow first and then extracting features.

[0167] As an optional embodiment, cleaning the historical vehicle message flow can include but is not limited to deduplication, verification, missing value processing and outlier processing.

[0168] In an example, deduplication can be deduplication of repeated vehicle message data in the historical vehicle message flow through message identifier (Identifier, ID) or timestamp.

[0169] In an example, the data integrity can be ensured by a verification algorithm.

[0170] The verification algorithm can be any one of existing verification algorithms such as parity check, cyclic redundancy check (Cyclic Redundancy Check, CRC) and the like, and the present application does not particularly limit this.

[0171] In an example, the missing value processing can be filling the vehicle message data with missing timestamp or regional data in the historical vehicle message flow with interpolation method or default value to preserve the integrity of the vehicle message data.

[0172] In an example, the outlier processing can be identifying the abnormal vehicle message data in the historical vehicle message flow by the method of Z-score, and performing smoothing processing or rejection to improve the accuracy of the historical vehicle message flow.

[0173] Further, in order to facilitate subsequent feature extraction and improve the efficiency of feature extraction, as an optional implementation, the cleaned historical vehicle message flow can be classified first, and then the features are extracted after classification.

[0174] The classification of the historical vehicle message flow can be that after collecting the historical vehicle message flow, the timestamp data set can be formed by collecting the timestamp of each vehicle message data in the historical vehicle message flow, the regional data set can be formed by collecting the regional information of each vehicle message data in the historical vehicle message flow, the vehicle type data set can be formed by collecting the vehicle type information of each vehicle message data in the historical vehicle message flow, and the message content data set can be formed by collecting the message content of each vehicle message in the historical vehicle message flow.

[0175] Among them, the timestamp dataset records the timestamp of the generation of each piece of historical vehicle message data, which can be used to analyze the time distribution characteristics of vehicle message flow. For example, the recording accuracy can be accurate to the millisecond level.

[0176] Among them, the regional dataset includes the geographical location of the vehicle, such as province, city, and region, which can be used to analyze the differences in vehicle message flow in different regions. Among them, the region in the geographical location refers to different administrative regions, such as rural areas, urban areas, and suburban areas, etc. For example, the analysis of the differences in vehicle message flow in different regions can be the differences in vehicle message flow in different cities of the same province, the differences in vehicle message flow in the same administrative region of different provinces, the differences in vehicle message flow in different administrative regions of the same city, etc.

[0177] Among them, the vehicle type dataset includes the vehicle type, such as commercial vehicles, passenger vehicles, and new energy vehicles, etc., which can be used to analyze the generation law of vehicle message flow of different types of vehicles.

[0178] Among them, the message content data includes the specific message content of each piece of historical vehicle message data, such as vehicle speed, engine state, fault code, etc., which can be used to assist in analyzing the characteristics of vehicle message flow.

[0179] Further, in some optional embodiments, extracting the time characteristics, regional characteristics, and vehicle type characteristics in the historical vehicle message flow includes:

[0180] S201B1, since the vehicle message flow has significant time correlation, the following time characteristics of the historical vehicle message flow can be extracted to obtain time characteristic data, which includes, for example:

[0181] 1) Hourly cycle: the change of vehicle message flow in different time periods of each day can be analyzed, for example, the vehicle message flow peak in the morning peak (7:00-9:00) and the evening peak (17:00-19:00) will be higher than that at other times.

[0182] 2) Daily cycle: the difference in vehicle message flow between weekdays and weekends can be analyzed, for example, the vehicle message flow during the weekday commuting period is significantly higher than that during the same period on weekends.

[0183] 3) Seasonal cycle: the change of vehicle message flow in different seasons can be analyzed, for example, the vehicle fault message may increase in winter.

[0184] 4) Holiday feature: mark statutory holidays such as the Spring Festival and the National Day, and analyze the change of vehicle message flow during the holidays.

[0185] 5) Special event features: the impact of extreme weather such as heavy rain, heavy snow, or large-scale activities such as car shows and sports events on vehicle message flow.

[0186] S201B2, due to the significant differences in vehicle density and driving habits in different regions, the following regional features of historical vehicle message flow can be extracted to obtain regional feature data, which exemplarily includes:

[0187] 1) Flow distribution: statistics of vehicle message flow in different city types or different regional types, for example, the vehicle message flow in first-tier cities is significantly higher than that in third- and fourth-tier cities. Among them, the regional type refers to different administrative regions, which can include but is not limited to urban areas, suburbs, and rural areas, etc.

[0188] 2) Traffic characteristics: analyze the differences in vehicle message flow between congested areas and non-congested areas.

[0189] S201B3, due to different types of vehicle message generation modes, the following vehicle type features of historical vehicle message flow can be extracted to obtain vehicle type feature data, which exemplarily includes:

[0190] 1) Vehicle type distribution: for example, the total amount of passenger cars, commercial vehicles, and new energy vehicles in the total vehicle message flow can be counted.

[0191] 2) Vehicle state features: analyze the impact of vehicle operating state (such as driving, idling, and engine off) on vehicle message generation frequency.

[0192] It can be understood that when extracting the time features of historical vehicle message flow, it can be directly extracted from the timestamp data; when extracting the regional features of historical vehicle message flow, it can be directly extracted from the regional data; when extracting the vehicle type of historical vehicle message flow, it can be directly extracted from the vehicle type data, so that each feature can be quickly obtained.

[0193] As an optional embodiment, the collected historical vehicle message flow can also be statistically analyzed, which can include:

[0194] 1) Mean and variance: exemplarily, the mean and variance of historical hourly or daily vehicle message flow can be counted, which can be used to measure the stability of historical vehicle message flow and provide a reference for predicting whether the future vehicle message flow is stable.

[0195] 2) Peak and valley: the peak and valley of historical vehicle message flow can be identified, which can provide a reference for subsequent prediction of whether the future vehicle message flow belongs to the peak or valley, and is beneficial to subsequent resource allocation.

[0196] 3) Growth rate: by calculating the change rate of historical vehicle message flow in adjacent time periods, the trend of future vehicle message flow can be predicted.

[0197] It can be understood that in an optional embodiment, the feature data obtained in step S201B can be directly output as a first feature data set to train a candidate model; in order to improve the training effect and make the obtained prediction model have better prediction results, in another optional embodiment, step S201C can be performed, and the feature data after step S201C is performed is output as a first feature data set to train a candidate model. The present application does not make special limitation thereon, and the subsequent example is taken as an example, that is, the feature data after step S201C is performed is output as a first feature data set to train a candidate model.

[0198] S201C, converting the feature data into machine language.

[0199] In some optional embodiments, converting the feature data into machine language can include but is not limited to feature encoding and feature standardization.

[0200] Different types of feature data can use different feature encoding methods.

[0201] For example, for regional feature data and vehicle type feature data, One-Hot encoding can be used, which can eliminate the error correlation of unordered category variables in the numerical process and ensure the accurate interpretation of discrete features by the machine learning model.

[0202] For example, for time feature data, the timestamp can be converted into a numerical feature, which can solve the storage, calculation and compatibility problems of the original time feature data.

[0203] All encoded feature data need to be standardized to eliminate the influence of dimension.

[0204] In order to facilitate subsequent training of candidate models and improve training efficiency, as an optional embodiment, the first feature data set can be output in the form of a table.

[0205] The first feature data set can include a time feature table, a regional feature table, a vehicle type feature table and a flow statistics table.

[0206] It can be understood that the time feature table is composed of time features extracted from the historical vehicle message flow, and can include fields such as timestamp, hour, day of the week, and whether it is a holiday.

[0207] It can be understood that the regional feature table is composed of regional features extracted from the historical vehicle message flow, and can include fields such as city identification (ID), area type, and area traffic proportion.

[0208] The city ID is also the area number, and each city corresponds to an identification, and different city identifications are different. The area type is also called the area type, which can include but is not limited to urban, suburban, and rural areas, etc. The area traffic proportion can also be referred to as the vehicle heat preservation traffic proportion of the area, that is, the ratio between the vehicle message flow of a certain area in a certain time period and the vehicle message flow of the whole country in the time period.

[0209] It can be understood that the vehicle type feature table is composed of vehicle type features extracted from the historical vehicle message flow, and can include fields such as vehicle type ID, vehicle state, and message generation frequency.

[0210] The vehicle type ID can also be referred to as the vehicle type number, that is, each vehicle corresponds to an identification, and the identifications of vehicles of different types are different. The message generation frequency is the frequency of generating a single vehicle message data.

[0211] It can be understood that the traffic statistics table is obtained by performing feature statistics on the collected historical vehicle message flow, and can include fields such as the mean, variance, peak value, valley value, and growth rate of the historical vehicle message flow.

[0212] It can be understood that the first feature data set is output in the form of a table as input for training the candidate model, which not only provides a large amount of data support for the training of the candidate model, but also improves the training efficiency.

[0213] S202, training a candidate model using the first feature data set to obtain a prediction model; so that the configuration of the cloud resource is dynamically adjusted according to the prediction result of the prediction model.

[0214] The prediction result is used to represent the size of the current vehicle message flow to be input to the cloud device.

[0215] The goal of training the prediction model is to analyze the historical vehicle message flow through a machine learning algorithm to build a high-precision prediction model that can capture the periodicity, trend, and burstiness of the historical vehicle message flow.

[0216] It can be understood that, since the vehicle message has significant time correlation, regional distribution characteristics and vehicle type difference, the time feature, the regional feature and the vehicle type feature can be determined based on the time information, the regional information and the vehicle type information of the historical vehicle message flow, the first feature data set is formed, data support can be provided for subsequent training of the candidate model, the inherent law of the data of the first feature data set can be captured, and the trained model can predict the trend of the vehicle message flow; by dynamically adjusting the configuration of the cloud resource according to the prediction result of the prediction model, the cloud resource can be fully utilized, and resource waste can be reduced.

[0217] In the embodiment of the application, the candidate model can include one or multiple. In order to improve the accuracy of prediction, the following example is taken as an example in which the candidate model includes multiple.

[0218] In the case where the candidate model includes multiple, the candidate model is trained by using the first feature data set to obtain a prediction model, including the following steps:

[0219] S202A, the first feature data set is filtered by using a correlation analysis algorithm and / or a feature importance evaluation algorithm to obtain a second feature data set.

[0220] It can be understood that the first feature data set obtained in step S201 has human subjective factors, that is, engineers think that what features have an impact on the prediction of the vehicle message flow, and extract what features. This extraction is not rational enough, therefore, in order to ensure rationality and improve scientificity, as an optional embodiment, the first feature data set needs to be filtered by using a correlation analysis algorithm and / or a feature importance evaluation algorithm to obtain a second feature data set. After the first feature data set is filtered by using the correlation analysis algorithm and / or the feature importance evaluation algorithm, each feature data obtained is the feature that has the greatest impact on the prediction of the vehicle message flow.

[0221] For example, the correlation analysis algorithm can be a Pearson coefficient correlation analysis algorithm; and the feature importance evaluation algorithm can be a tree model-based feature importance sorting method.

[0222] For example, the time feature included in the obtained second feature data set can include hours, days of the week and whether it is a holiday; the regional feature included in the obtained second feature data set can include regional types and regional flow proportions; and the vehicle type feature included in the obtained second feature data set can include vehicle type distribution and vehicle state.

[0223] S202B, each candidate model in the multiple candidate models is trained by using the second feature data set.

[0224] In an alternative embodiment, a time series division strategy can be adopted to divide the second feature dataset into a training set, a validation set and a test set, so as to ensure the continuity of the training set, the validation set and the test set in time, and avoid data omission or leakage. It can be understood that the training set is used to train each candidate model, the validation set is used to evaluate the effect of the hyperparameters (such as learning rate and network layer number, etc.) of each candidate model, and the test set is used to evaluate the final performance (such as prediction accuracy and inference time consumption) of each prediction model.

[0225] Exemplarily, 70% of the second feature dataset is divided into a training set, 15% of the second feature dataset is divided into a validation set, and 15% of the second feature dataset is divided into a test set. It can be understood that the above 70%, 15% and 15% are examples of how much the training set, the validation set and the test set are divided, and are not limited. In actual application, they can be other numerical values, and the present application does not make special limitation thereto, as long as the data of the training set is greater than the sum of the data of the validation set and the data of the test set.

[0226] Exemplarily, the candidate models can be divided into three categories, the first category includes traditional time series models, the second category includes tree models, and the third category includes deep learning models.

[0227] Further, exemplarily, the traditional time series model can include an autoregressive integrated moving average model (ARIMA) and a seasonal autoregressive integrated moving average model (SARIMA). It can be understood that in actual application, the traditional time series model can include more or one.

[0228] Further, exemplarily, the tree model can include an eXtreme Gradient Boosting (XGBoost) and a Light Gradient Boosting Machine (LightGBM), which are suitable for capturing nonlinear feature interactions. It can be understood that in actual application, the tree model can include more or one.

[0229] Further, exemplarily, the deep learning model can include a Long Short-Term Memory (LSTM) and a Transformer, which are suitable for processing long-term dependencies. It can be understood that in actual application, the deep learning model can include more or one.

[0230] To make the trained candidate models have better prediction accuracy, as another optional embodiment, the hyperparameters of each candidate model can also be optimized and loss function can be added during the training of each candidate model.

[0231] Exemplarily, the hyperparameters of each prediction model can be optimized by grid search or Bayesian optimization. For example, the number of layers, the number of hidden units and the learning rate of LSTM can be optimized; the tree depth, the learning rate and the subsampling ratio of XGBoost can be optimized.

[0232] Exemplarily, the loss function can be Mean Squared Error (MSE) or Mean Absolute Error (MAE). It can be understood that the loss function is added to evaluate the training effect and see whether the difference between the predicted value and the target value is too large.

[0233] S202C, the prediction accuracy and real-time performance of the trained candidate models are evaluated, and a candidate model meeting a preset evaluation condition is selected as a prediction model.

[0234] Exemplarily, the prediction accuracy of the trained candidate models can be evaluated by at least one of the following: for example, MSE, MAE and score.

[0235] Among them, for when evaluating the prediction accuracy of each candidate model, The score value range is usually [0, 1], and the The closer the score of the candidate model is to 1, the stronger the explanation ability of the candidate model is.

[0236] Exemplarily, the real-time performance refers to the inference time consumption of each candidate model.

[0237] In an optional embodiment, the preset evaluation condition is related to the evaluation method selected when evaluating the prediction accuracy of the trained candidate models and the real-time performance.

[0238] Exemplarily, when the evaluation method is to evaluate by MSE, the preset evaluation condition can be that both the minimum MSE and the shortest inference time consumption are met, and the preset evaluation condition can also be the best comprehensive performance (i.e. the MSE is small and the inference time consumption is short). Exemplarily, from the candidate models, the candidate model with the minimum MSE and the shortest inference time consumption can be selected as the prediction model for prediction, and if there is no candidate model that can meet both the minimum MSE and the shortest inference time consumption in the candidate models, the candidate model with the small MSE and the short inference time consumption can be selected as the prediction model.

[0239] For example, when the evaluation manner is MAE evaluation, the preset evaluation condition can be that both the MAE minimum and the reasoning time consumption minimum are met, and the preset evaluation condition can also be that the comprehensive performance is optimal (i.e., the MAE is small and the reasoning time consumption is short). For example, from the candidate models, the candidate model with the minimum MAE and the minimum reasoning time consumption can be preferentially selected as the prediction model for prediction, and if there is no candidate model that can meet both the minimum MAE and the minimum reasoning time consumption in the candidate models, the candidate model with the small MAE and the short reasoning time consumption can be selected as the prediction model.

[0240] For example, when the evaluation manner is score evaluation, the preset evaluation condition can be that both the maximum score and the minimum reasoning time consumption are met, and the preset evaluation condition can also be that the comprehensive performance is optimal (i.e., the score is large and the reasoning time consumption is short). For example, from the candidate models, the candidate model with the maximum score and the minimum reasoning time consumption can be preferentially selected as the prediction model for prediction, and if there is no candidate model that can meet both the maximum score and the minimum reasoning time consumption in the candidate models, the candidate model with the large score and the short reasoning time consumption can be selected as the prediction model. For example, when the evaluation manner is MSE, MAE and score evaluation, the preset evaluation condition can be that both the MSE and MAE minimum and the maximum score are met, and the preset evaluation condition can also be that the comprehensive performance is optimal (i.e., the MSE and MAE are small, and the score is large). For example, from the candidate models, the candidate model with the minimum MSE, the minimum MAE, the maximum score and the minimum reasoning time consumption can be preferentially selected as the prediction model for prediction, and if there is no candidate model that can meet both the minimum MSE, the minimum MAE, the maximum score and the minimum reasoning time consumption in the candidate models, the candidate model with the small MSE, the small MAE, the large score and the short reasoning time consumption can be selected as the prediction model.

[0241] For example, when the evaluation manner is MSE, MAE and score evaluation, the preset evaluation condition can be that both the MSE and MAE minimum and the maximum score are met, and the preset evaluation condition can also be that the comprehensive performance is optimal (i.e., the MSE and MAE are small, and the score is large). For example, from the candidate models, the candidate model with the minimum MSE, the minimum MAE, the maximum score and the minimum reasoning time consumption can be preferentially selected as the prediction model for prediction, and if there is no candidate model that can meet both the minimum MSE, the minimum MAE, the maximum score and the minimum reasoning time consumption in the candidate models, the candidate model with the small MSE, the small MAE, the large score and the short reasoning time consumption can be selected as the prediction model. For example, when the evaluation manner is MSE, MAE and score evaluation, the preset evaluation condition can be that both the MSE and MAE minimum and the maximum score are met, and the preset evaluation condition can also be that the comprehensive performance is optimal (i.e., the MSE and MAE are small, and the score is large). For example, from the candidate models, the candidate model with the minimum MSE, the minimum MAE, the maximum score and the minimum reasoning time consumption can be preferentially selected as the prediction model for prediction, and if there is no candidate model that can meet both the minimum MSE, the minimum MAE, the maximum score and the minimum reasoning time consumption in the candidate models, the candidate model with the small MSE, the small MAE, the large score and the short reasoning time consumption can be selected as the prediction model.

[0242] ​​​​​​In order to make the determined prediction model have higher accuracy and robustness, S202C can also be, as another optional embodiment, evaluating the prediction accuracy and real-time performance of each trained candidate model, and determining the prediction model according to each trained candidate model and the weight corresponding to each candidate model, so that each candidate model can be used to cooperatively predict the vehicle message flow, and the prediction performance can be significantly improved.

[0243] The weight corresponding to each candidate model is related to the prediction accuracy and real-time performance of each candidate model.

[0244] In an optional embodiment, determining the prediction model according to each trained candidate model and the weight corresponding to each candidate model includes:

[0245] First, the accuracy weight corresponding to the prediction accuracy of each candidate model and the real-time weight corresponding to the real-time performance of each candidate model are determined.

[0246] The accuracy weight of each candidate model can be the same or different, and the accuracy weight of each candidate model can be dynamically adjusted according to the performance of each candidate model, for example, the accuracy weight of a candidate model with high prediction accuracy is high, and the accuracy weight of a candidate model with low prediction accuracy is low. The real-time weight of each candidate model can be the same or different, and the real-time weight of each candidate model can be dynamically adjusted according to the performance of each candidate model, for example, the real-time weight of a candidate model with high real-time performance (i.e. long inference time) is low, and the real-time weight of a candidate model with low real-time performance (i.e. short inference time) is high.

[0247] Then, the product corresponding to each candidate model is determined.

[0248] The product corresponding to each candidate model refers to the value obtained by multiplying each candidate model by the sum of the accuracy weight and the real-time weight corresponding to each candidate model.

[0249] Finally, the products corresponding to each candidate model are summed to obtain the prediction model.

[0250] It can be understood that through the above embodiments, two acquisition methods of the prediction model are provided, one is to select one from the plurality of trained candidate models as the prediction model, or to integrate the plurality of candidate models to obtain the prediction model. By screening the first feature data set by using the correlation analysis algorithm and / or the importance evaluation algorithm of the feature, the second feature data set is obtained, and each feature data in the obtained second feature data set is more significant feature data for the prediction of the prediction model. Training each candidate model in the plurality of candidate models by using the second feature data set can make the prediction of each candidate model for the vehicle message flow more accurate; by evaluating the prediction accuracy and real-time performance of each trained candidate model, the accuracy of the prediction of each candidate model for the vehicle message flow and the time length required to obtain the prediction result can be obtained; when one is selected from the plurality of trained candidate models as the prediction model: by using the preset evaluation condition to select from each candidate model, the selected candidate model can be more suitable for the expected prediction of the vehicle message flow to adapt to the working condition of limited computing resources; when the plurality of candidate models are integrated to obtain the prediction model: the prediction model is determined according to the trained each candidate model and the weight corresponding to each candidate model, so that the determined prediction model has higher accuracy and robustness, and each candidate model cooperates to predict the vehicle message flow, which can significantly improve the prediction performance.

[0251] As another optional embodiment, the prediction model trained by the embodiments of the application further includes an online learning function.

[0252] Exemplarily, the online learning function can include incremental training and a sliding window mechanism.

[0253] Among them, the incremental learning can be to input the newly added traffic data into the model periodically for incremental training to adapt to the change of traffic mode.

[0254] Exemplarily,

[0255] Among them, the sliding window mechanism can use data in a fixed time window as a training set to ensure that the model always learns the latest mode.

[0256] As another optional embodiment, the prediction model trained by the embodiments of the application further includes a feedback optimization function.

[0257] Exemplarily, the feedback optimization function can include prediction error monitoring and abnormal traffic detection.

[0258] Among them, the prediction error monitoring can be to calculate the error between the predicted value and the actual value in real time, and to trigger model retraining when the error exceeds the threshold.

[0259] The abnormal traffic detection can be performed by an isolation forest algorithm or a local outlier factor (LOF) algorithm to detect abnormal traffic and dynamically adjust model parameters.

[0260] As another optional embodiment, the prediction model trained by the embodiments of the present application further includes fault recovery and consistency guarantee.

[0261] For example, a multi-level fault recovery mechanism is designed to deal with network interruption or cluster anomaly. When a synchronization failure is detected, the system first attempts to automatically retry, and gradually extends the retry interval to avoid the snowball effect. If the retry fails, a local cache mechanism is triggered to temporarily store data to a disk queue, and then re-synchronize after the network is restored.

[0262] To ensure data consistency, the Kafka transaction mechanism is used to achieve the Exactly-Once semantics, and the periodic offset check is used to prevent data loss or duplication. In addition, the system records synchronization state checkpoints (Checkpoint) to support fast recovery from breakpoints and reduce resource consumption caused by full synchronization.

[0263] The following examples describe possible implementation schemes of the processing method of the cloud resource and the training method of the prediction model according to one or more embodiments described above.

[0264] Cloudera Distribution for Hadoop (CDH): an enterprise-level big data platform deployed based on a cloud computing environment, which combines the distributed computing (Hadoop) ecological components of traditional CDH with the elastic resource management capabilities of cloud services, providing a big data solution with on-demand expansion, automated operation and maintenance, and hybrid cloud support.

[0265] Cloudera Data Platform (CDP): an enterprise-level big data platform deployed locally by Cloudera, which mainly runs on user-owned private data centers or local servers, providing complete big data management capabilities.

[0266] Hot data: refers to data that is accessed frequently within a certain time period, has strong timeliness, and is crucial to business operation, which requires high-performance storage to achieve fast read-write response.

[0267] Application Programming Interface (API): API is a set of predefined functions or protocols for interaction and data sharing between different software systems, and developers can call functions without understanding the internal implementation details.

[0268] Controller Area Network (CAN): CAN is a widely used serial communication protocol in the field of automotive electronics, industrial control, etc., mainly used for real-time data exchange between devices. Its core features include high reliability and anti-interference capability, commonly used for communication between sensors, actuators and controllers.

[0269] The scheme provided by the embodiment belongs to the technical field of Internet of Vehicles data synchronization, and specifically relates to a dynamic adjustment method of cloud resources based on a prediction model, and is especially suitable for vehicle message data synchronization scenarios between public clouds and private clouds.

[0270] With the rapid development of Internet of Vehicles technology and the popularity of intelligent vehicles, the CAN bus data generated by vehicles (i.e. vehicle message data) presents an explosive growth, reaching hundreds of billions per day. These data are crucial for vehicle state monitoring, driving behavior analysis, intelligent traffic management and other applications. Currently, the storage and analysis of Internet of Vehicles data mainly adopt a combined architecture mode of cloud (public cloud) and cloud (private cloud): the cloud CDH big data cluster is responsible for hot data storage, while the cloud CDP big data cluster undertakes the storage and deep analysis tasks of full data. Under this architecture, the core process of data synchronization is to pull data from the cloud Kafka cluster to the cloud Kafka cluster, and then complete the subsequent processing. However, the related technical scheme has the following significant defects when dealing with the massive data synchronization needs of Internet of Vehicles scenarios:

[0271] 1. Difficult to cope with data volume and high real-time requirement application scenarios:

[0272] In the Internet of Vehicles scenario, with the continuous growth of automobile sales and the continuous improvement of vehicle intelligence, the vehicle message data generated by modern vehicles per day has reached hundreds of billions. These data not only contain vehicle running status, sensor information, but also involve high real-time business data such as intelligent driving, vehicle interaction. Due to the high frequency and high concurrency characteristics of vehicle message data, the traditional batch processing data transmission mode is difficult to meet the real-time analysis needs, so it is necessary to rely on Kafka and other distributed message queues to realize low-latency data synchronization. However, in the cross-cloud data synchronization process, the current Kafka cluster often causes message backlog due to the surge in data volume, and even causes consumer delay, seriously affecting the timeliness of downstream data analysis. In addition, the real-time requirement of vehicle networking business for data synchronization is very high, for example, automatic driving data analysis, remote fault diagnosis and other scenarios all need millisecond-level response, and the existing synchronization mechanism is difficult to maintain stable low-latency transmission during data flood.

[0273] 2. There are processing defects during data flood:

[0274] The related Kafka cross-cloud data synchronization scheme usually adopts a static resource configuration strategy, that is, a fixed computing resource (such as a throughput node, a partition number, a consumer group instance, etc.) is pre-allocated, and when facing a sudden vehicle message flow (that is, the total of a plurality of vehicle message data), it cannot be dynamically adjusted according to the actual flow fluctuation. For example, during the morning and evening peak traffic periods, the amount of vehicle message data may increase several times, and the vehicle message flow becomes large, while the current cross-cloud resource lacks automatic expansion and contraction capability, resulting in insufficient processing capacity of the Kafka cluster on the cloud, and serious message backlog. In addition, when the amount of vehicle message data falls (that is, decreases), the vehicle message flow becomes small, and the fixed resource allocation will cause idle waste of computing resources, increasing the cost of cloud services. Although some schemes attempt to adjust resources manually to cope with flow changes, due to the lack of accurate flow prediction capability, manual intervention often lags behind actual demand, and cannot expand in advance before the data flood arrives, resulting in persistent synchronization delay problems.

[0275] 3. Lack of cross-cloud resource coordination:

[0276] In the Internet of Vehicles data architecture, cross-cloud resources include Kafka on the cloud and Kafka off the cloud. The cloud environment (public cloud) is usually used to store and process hot data, while the cloud environment (private cloud) is responsible for long-term storage and deep analysis of full data. The current Kafka resource allocation on the cloud (CDH cluster) and off the cloud (CDP cluster) is usually static or manually adjusted, lacking an automatic cross-cloud resource coordination mechanism, resulting in uneven cross-cloud resource allocation. For example, when the Kafka cluster on the cloud is overloaded due to an increase in the amount of vehicle message data, the off-cloud consumers may not be able to consume data in a timely manner due to insufficient resources, and vice versa. In addition, there is competition for resources such as network bandwidth, storage input (Input, I) / output (Output, O) between different cloud environments, and the current synchronization strategy cannot dynamically optimize the network state and cluster load, which can easily cause data transmission bottlenecks. Although some research attempts to optimize Kafka consumer allocation through load balancing technology, it still does not solve the core problem of cross-cloud resource coordination scheduling, resulting in low resource utilization and unstable synchronization efficiency.

[0277] 4. Lack of dynamic flow prediction capability:

[0278] Most related Kafka cross-cloud synchronization schemes adopt a passive response resource adjustment strategy, that is, resource expansion is triggered only after monitoring the increase of message backlog or delay. This approach has obvious hysteresis. Since vehicle message flow has strong spatiotemporal regularity (such as morning and evening peak, holiday travel peak, etc.), if historical data is combined for flow prediction, resource allocation can be optimized in advance to avoid sudden traffic impact. However, current vehicle networking data synchronization systems generally lack dynamic flow prediction capability based on machine learning, and cannot accurately predict the amount of vehicle message data in the future period, resulting in a passive state of resource adjustment. In addition, vehicle data flow may also be affected by sudden events (such as severe weather and large-scale activities, etc.), and traditional static models are difficult to adapt to such dynamic changes, further exacerbating the irrationality of resource allocation.

[0279] I. Technical problems to be solved by the embodiment:

[0280] In the vehicle networking scenario, the amount of CAN message data generated by vehicles (i.e., the amount of vehicle message data) grows exponentially, and the data synchronization between Kafka clusters on the cloud (public cloud) and off the cloud (private cloud) faces the following key challenges:

[0281] 1. Large data volume and high real-time requirement: Vehicles generate hundreds of billions of CAN message data (i.e., vehicle message data) every day, and traditional static resource allocation methods cannot adapt to sudden traffic, resulting in data synchronization delay or loss.

[0282] 2. Data flood processing defects: Related Kafka cross-cloud synchronization mechanisms lack the ability to predict traffic fluctuations and cannot adjust resources in advance, resulting in message backlog during traffic flood and affecting data processing timeliness.

[0283] 3. Insufficient cloud resource coordination: Public cloud Kafka clusters and private cloud Kafka clusters have uneven resource allocation, which may result in one side being overloaded while the other side being idle, causing resource waste or synchronization bottlenecks.

[0284] 4. Lack of dynamic flow prediction capability: Related solutions usually use fixed resource configuration or simple threshold triggering for expansion and contraction, and cannot intelligently predict based on historical traffic patterns, resulting in lagging resource adjustment.

[0285] The embodiment aims to solve the above problems and proposes a dynamic adjustment method for cloud resources based on a prediction model, which predicts traffic trends through machine learning and combines Kafka cluster dynamic expansion and contraction strategies to achieve optimal allocation of cloud resources, improve data synchronization efficiency, and reduce resource waste.

[0286] II. Technical scheme of the embodiment:

[0287] The embodiment proposes a dynamic adjustment method of cloud resources based on a prediction model, aiming to solve the problems of uneven resource allocation, synchronization delay and insufficient data flood processing capacity faced in the cross-cloud synchronization of massive vehicle message data in the Internet of Vehicles scenario. The method realizes intelligent collaborative allocation of resources of public cloud Kafka cluster and private cloud Kafka cluster by combining traffic prediction model and dynamic resource scheduling strategy, improves data synchronization efficiency and reduces resource waste. The core steps of the embodiment are as follows:

[0288] 1. Data collection and feature extraction.

[0289] Collect historical vehicle message traffic data and store it classified by timestamp, region and vehicle type. Extract periodic features (such as morning and evening rush hours) and event features (such as holidays and extreme weather) through feature engineering to provide high-value input for the prediction model.

[0290] 2. Train the prediction model.

[0291] Based on historical vehicle message traffic, use time series analysis and machine learning algorithms to build a prediction model to identify the periodicity and burstiness of traffic changes. At the same time, introduce real-time monitoring and feedback mechanism to dynamically adjust the parameters of the prediction model to adapt to the real-time changes of vehicle message traffic and improve the prediction accuracy.

[0292] 3. Dynamic resource allocation strategy.

[0293] Based on the prediction results of the prediction model, dynamically adjust the resources of Kafka cluster to realize efficient collaboration across cloud environments. Through cloud platform API, elastically expand and shrink the number of Kafka Broker nodes to ensure that resource supply matches traffic demand; automatically scale consumer group instances to avoid delay caused by message accumulation; when the predicted traffic exceeds the current partition processing capacity, call Kafka Admin API to dynamically add partitions to improve parallel processing efficiency. At the same time, set different priorities for different business data (such as emergency alarm messages and regular status data) to prioritize the synchronization of high-timeliness data and avoid low-priority data occupying critical resources.

[0294] 4. Data synchronization optimization.

[0295] Through intelligent scheduling algorithm, dynamically select the optimal data transmission path (such as private line or public network), combine real-time network status, data traffic and resource load situation to reduce cross-cloud synchronization delay; apply load balancing technology between Kafka clusters to ensure even distribution of data and avoid single node overload to improve overall system stability. In addition, through real-time monitoring of synchronization progress and resource utilization, dynamically adjust data transmission strategy to further optimize synchronization efficiency and ensure efficient and reliable transmission of massive vehicle messages in cross-cloud environment.

[0296] Thirdly, the embodiment provides a specific process of the scheme.

[0297] Figure 3 A flowchart of a dynamic adjustment method of cloud resources based on a prediction model provided by the embodiment of the application is shown in Figure 3 The method comprises the following steps:

[0298] S301, data collection and feature extraction.

[0299] The data collection and feature extraction are completed by a data collection and feature processing module, which is a core basic module of the embodiment. The main target is to collect historical vehicle message data and extract key features therefrom, thereby providing high-quality data support for subsequent prediction model training. In the Internet of Vehicles scenario, vehicle message data has significant time correlation, regional distribution characteristics and vehicle type differences. Therefore, the module needs to classify, clean and extract features from the original data from multiple dimensions to capture the inherent rules of the data.

[0300] S301A, data collection.

[0301] S301A1, data source: the data source of the data collection and feature processing module mainly includes Kafka clusters in the cloud environment (public cloud) and the cloud environment (private cloud).

[0302] The specific collected data includes:

[0303] Timestamp data: records the timestamp of generation of each vehicle message, accurate to millisecond level, and is used for analyzing the time distribution characteristics of traffic.

[0304] Regional data: includes the geographical location to which the vehicle belongs, such as province, city and region, and is used for analyzing the traffic difference of different regions.

[0305] Vehicle type data: distinguishes vehicle types, such as passenger cars, commercial vehicles and new energy vehicles, and is used for analyzing the message generation rule of different types of vehicles.

[0306] Message content data: includes the specific content of the vehicle message, such as vehicle speed, engine state and fault code, and is used for assisting traffic feature analysis.

[0307] S301A2, data collection method.

[0308] The data collection is realized by the following technologies:

[0309] Kafka consumer group subscription: deploy a consumer group in the cloud and cloud Kafka cluster, subscribe to vehicle message data in real time, and persist the data to a distributed storage system HDFS.

[0310] Timed batch pulling: To avoid the impact of real-time collection on system performance, a timed batch pulling strategy is adopted, such as pulling data every 5 minutes.

[0311] Data deduplication and verification: Duplicate data is deduplicated by message ID or timestamp, and the integrity of the data is ensured by a verification algorithm.

[0312] S301B, feature extraction.

[0313] Among them, the feature extraction includes time dimension feature (i.e. time feature), regional dimension feature (i.e. regional feature), vehicle type dimension feature (i.e. vehicle type feature) and traffic statistics feature.

[0314] S301B1, time dimension feature extraction is performed.

[0315] Vehicle message traffic has significant time correlation, so the following time dimension features need to be extracted:

[0316] Hourly cycle: Analyze the traffic changes in different time periods of each day, such as the traffic peak in the morning rush hour (e.g. 7:00-9:00) and the evening rush hour (e.g. 17:00-19:00).

[0317] Daily cycle: Analyze the traffic difference between weekdays and weekends, for example, the traffic during the weekday commuting period is significantly higher than that during the weekend.

[0318] Seasonal cycle: Analyze the traffic changes in different seasons, for example, the number of vehicle fault messages may increase in winter.

[0319] Holiday feature: Mark statutory holidays such as the Spring Festival and National Day, and analyze the traffic changes during the holidays.

[0320] Special event feature: The impact of extreme weather such as heavy rain and snow, or large-scale activities such as car shows and sports events on traffic.

[0321] S301B2, regional dimension feature extraction is performed.

[0322] The vehicle density and driving habits of different regions are significantly different, and the following regional features need to be extracted:

[0323] Traffic distribution: Statistics of message traffic in different cities or regions, for example, the traffic in first-tier cities is significantly higher than that in third and fourth-tier cities.

[0324] Traffic feature: Analyze the traffic difference between congested areas and non-congested areas.

[0325] S301B3, vehicle type dimension feature extraction is performed.

[0326] Different types of vehicle message generation patterns are different, and the following features need to be extracted:

[0327] Vehicle type distribution: Statistics of the proportion of messages of passenger cars, commercial vehicles, new energy vehicles, etc.

[0328] Vehicle state characteristics: Analysis of the influence of vehicle running state (such as driving, idling, engine off) on message generation frequency.

[0329] S301B4, extraction of flow statistics features.

[0330] Based on the original data, the following statistical features are calculated:

[0331] Mean and variance: Statistics of the mean and variance of the flow per hour or per day, used to measure the stability of the flow.

[0332] Peak and valley: Identify the peak and valley of historical flow, provide reference for resource allocation.

[0333] Growth rate: Calculate the flow change rate of adjacent time periods, used to predict future flow trends.

[0334] S301C, data preprocessing.

[0335] Among them, data preprocessing includes data cleaning, feature encoding and feature standardization.

[0336] S301C1, data cleaning.

[0337] Missing value processing: For missing time stamp or region data, use interpolation method or default value to fill.

[0338] Outlier processing: Identify abnormal flow data by Z-Score method, and smooth or remove.

[0339] S301C2, feature encoding.

[0340] Specifically, it includes category feature encoding and time feature encoding.

[0341] Among them, category feature encoding: One-Hot encoding of regional, vehicle type and other category features.

[0342] Among them, time feature encoding: Convert time stamp to numerical feature.

[0343] S301C3, feature standardization.

[0344] Standardize each feature after encoding to eliminate dimension influence.

[0345] S301D, data acquisition and feature processing module output: The final output of the data acquisition and feature extraction module is a structured feature dataset (i.e. the first feature dataset).

[0346] The structured feature dataset includes the following:

[0347] Time feature table: Contains fields such as timestamp, hour, day of the week, whether it is a holiday, etc.

[0348] Regional feature table: Contains fields such as region ID, region type (such as urban, suburban, etc.), and region traffic proportion.

[0349] Vehicle type feature table: Contains fields such as vehicle type ID, vehicle status, and message generation frequency.

[0350] Traffic statistics table: Contains fields such as traffic mean, peak value, and growth rate.

[0351] The output will serve as the input for the prediction model, providing data support for subsequent dynamic resource allocation.

[0352] S302, train the prediction model.

[0353] The training of the prediction model is completed by the training module. The training module is one of the core modules of the embodiment, and its goal is to analyze historical vehicle message traffic through machine learning algorithms to build a high-precision prediction model. This module can capture the periodicity, trend, and burstiness of traffic and support real-time adjustments to adapt to dynamically changing traffic patterns. Based on the prediction results, the system can optimize the resource configuration of the Kafka cluster in advance to avoid resource waste or synchronization delay.

[0354] S302A, historical data analysis.

[0355] The prediction model is trained by analyzing historical data, including the following steps:

[0356] S302A1, data preparation.

[0357] The data preparation work includes dividing the data set and filtering the structured feature dataset.

[0358] S302A1a, divide the data set.

[0359] Divide the structured data output by the feature extraction module into a training set (e.g., 70%), a validation set (e.g., 15%), and a test set (e.g., 15%).

[0360] Use a time series division strategy to ensure the time continuity of the training set and the test set, and avoid future data leakage.

[0361] S302A1b, filter the structured feature dataset.

[0362] A second feature dataset is obtained by screening the structured feature dataset.

[0363] The features that significantly affect the traffic prediction can be screened through Pearson coefficient correlation analysis and feature importance evaluation based on tree model feature importance ranking, for example:

[0364] Time features: hour, day of the week, whether it is a holiday.

[0365] Regional features: regional traffic proportion, regional type.

[0366] Vehicle type features: vehicle type distribution, vehicle status.

[0367] S302A2, model selection and training.

[0368] S302A2a, determine candidate models.

[0369] For the time series characteristics of vehicle message traffic, the following machine learning models are selected for comparative experiments.

[0370] Traditional time series models: ARIMA and SARIMA.

[0371] Tree models: XGBoost and LightGBM, suitable for capturing nonlinear feature interactions.

[0372] Deep learning models: LSTM and Transformer, suitable for handling long-term dependencies.

[0373] S302A2b, model training.

[0374] Parameter tuning: tune model hyperparameters through grid search or Bayesian optimization. For example:

[0375] The number of layers of LSTM, the number of hidden units, the learning rate; the tree depth of XGBoost, the learning rate, and the subsampling ratio.

[0376] Loss function: use mean square error (MSE) or mean absolute error (MAE) as the loss function.

[0377] S302A2c, model evaluation.

[0378] Evaluate the performance of each candidate model on the validation set and test set, the main indicators are as follows, select the candidate model with the best comprehensive performance as the final prediction model.

[0379] Prediction accuracy: MSE, MAE, Score.

[0380] Real-time performance: model inference time.

[0381] Wherein, the detailed description of the optimal comprehensive performance can refer to the foregoing related expression, which will not be repeated here.

[0382] As another optional embodiment, model integration can be performed, and the integrated total model is taken as the final prediction model. The following steps S302A2d can be referred to.

[0383] S302A2d, model integration.

[0384] To improve the prediction robustness, a weighted average method can be used for model integration, and the prediction results of multiple candidate models are weighted and fused, and the weight is dynamically adjusted according to the model performance.

[0385] S302B, real-time adjustment mechanism.

[0386] Wherein, the real-time adjustment mechanism includes online learning and feedback optimization.

[0387] S302B1, online learning.

[0388] Incremental training: periodically input new traffic data into the prediction model for incremental training to adapt to the changes of traffic patterns.

[0389] Sliding window mechanism: use fixed time window data as the training set to ensure that the prediction model always learns the latest pattern.

[0390] S302B2, feedback optimization.

[0391] Prediction error monitoring: real-time calculation of the error between the predicted value and the actual value, and model retraining is triggered when the error exceeds the threshold.

[0392] Abnormal traffic detection: detect abnormal traffic through isolation forest or LOF (local outlier factor) algorithm, and dynamically adjust the model parameters.

[0393] S303, dynamic resource allocation strategy.

[0394] Wherein, the dynamic resource allocation strategy is executed by a dynamic resource allocation module. This module is the core execution module of the embodiment, and its goal is to adjust the resource configuration of the Kafka cluster in real time according to the prediction results output by the prediction model, including the number of Broker instances, the number of partitions, the size of consumer groups, etc. to cope with the fluctuations of vehicle message traffic. Through elastic scaling, priority management and load balancing technology, the module ensures the efficiency and stability of cross-cloud data synchronization, while avoiding resource waste.

[0395] S303A, elastic resource configuration.

[0396] Wherein, the elastic resource configuration includes throughput node (Broker) dynamic scaling.

[0397] The Broker scaling trigger condition is:

[0398] (1) When the prediction result exceeds 80% of the current Broker processing capacity (i.e., exceeds the first processing capacity of the cloud resources), trigger the scaling operation to add a Broker instance (i.e., increase the throughput node).

[0399] (2) When the prediction result is less than 30% of the current Broker processing capacity (i.e., is less than the second processing capacity of the cloud resources), trigger the scaling operation to reduce the Broker instance (i.e., reduce the throughput node).

[0400] The way to achieve Broker scaling is to call the cloud platform API to dynamically adjust the number of Brokers.

[0401] By way of example, the blue-green deployment strategy is adopted to ensure that the service does not interrupt during the scaling process.

[0402] S303B, dynamic adjustment of partitions.

[0403] The dynamic adjustment of partitions includes partition scaling and partition scaling.

[0404] The trigger condition for dynamic adjustment of partitions is:

[0405] (1) When the vehicle message flow of a single partition (i.e., the first area) exceeds the first threshold, determine the increase in vehicle message flow, and add partitions to the partition according to the increase in vehicle message flow.

[0406] The Kafka Admin API can be called to add partitions.

[0407] By way of example, the number of added partitions is allocated in proportion to the predicted flow (e.g., if the flow increases by 20%, 20% of the partitions are added).

[0408] It can be understood that since the vehicle message data volume is continuously generated, if the consumer in the cloud resources cannot consume (i.e., process) these generated vehicle message data volume in time, information accumulation will occur. Therefore, as another optional embodiment, when the message accumulation amount of the first area exceeds the first preset accumulation amount, it is determined that the vehicle message flow of a single partition (i.e., the first area) exceeds the first threshold.

[0409] (2) When the prediction result indicates that the vehicle message flow is in a trough period, merge low-activity partitions to reduce resource occupation.

[0410] S303C, consumer group management.

[0411] S303C1, scaling of consumer instances.

[0412] The triggering condition for scaling of consumer instances is:

[0413] (1) When the prediction result is greater than the second threshold, increase the consumer instances.

[0414] (2) When the prediction result is less than the second threshold and exceeds 10 minutes, decrease the consumer instances.

[0415] It can be understood that, since the vehicle message data volume is continuously generated, if the consumer in the cloud resource cannot timely consume (i.e., process) the generated vehicle message data volume, information accumulation will occur. Therefore, as another optional embodiment, the consumer instances can be automatically increased or decreased based on the message accumulation (Lag).

[0416] That is, when the Lag exceeds the second preset accumulation, it is determined that the prediction result is greater than the second threshold, and the consumer instances are increased. When the Lag continuously falls below the second preset accumulation and exceeds 10 minutes, the prediction result is less than the second threshold and exceeds 10 minutes, and the consumer instances are decreased.

[0417] The way to achieve scaling of consumer instances is to monitor the Lag through the Kafka Consumer Group API, and adjust the number of consumer instances in combination with the container orchestration tool.

[0418] S303C2, consumer load balancing.

[0419] When the consumer instances are increased or decreased, partition rebalancing (Rebalance) is triggered to ensure load balancing of the consumers. That is, when the consumer instances are increased, the partitions in the area where the consumer instances are located are also increased; when the consumer instances are decreased, the partitions in the area where the consumer instances are located are also decreased.

[0420] The StickyAssignor distribution strategy of Kafka can be used to reduce the performance overhead caused by Rebalance.

[0421] S303D, priority management.

[0422] S303D1, first divide the priority of the vehicle message.

[0423] The vehicle message is divided into three priorities according to the business importance, i.e., high, medium, and low. The fault alarm message is a high-priority message (i.e., a vehicle message of high importance level), and the regular state message is a low-priority message (i.e., a vehicle message of low importance level). That is, the vehicle message including the fault alarm information is regarded as a vehicle message of high importance level, and the vehicle message including the regular state information is regarded as a vehicle message of low importance level.

[0424] S303D2, Resource Allocation Strategy.

[0425] High-priority messages are allocated to exclusive brokers or partitions (i.e., independent areas) to ensure low-latency processing.

[0426] Low-priority messages are transmitted using batch compression to reduce resource consumption.

[0427] S303E, data synchronization optimization.

[0428] Data synchronization optimization aims to improve the efficiency and stability of data transmission between cross-cloud Kafka clusters through intelligent scheduling algorithms and load balancing technology. Based on traffic prediction results and real-time resource status, data synchronization optimization dynamically optimizes data transmission paths, compression strategies, and fault recovery mechanisms to ensure efficient synchronization of vehicle message data between cloud and on-premises environments.

[0429] S303E1, Intelligent Scheduling Algorithm.

[0430] The intelligent scheduling algorithm dynamically selects the optimal data transmission path by comprehensively analyzing network status, data traffic characteristics, and resource load.

[0431] First, the algorithm collects network latency, bandwidth utilization, and node load metrics between on-premises and cloud-based Kafka clusters in real time to build a multi-dimensional evaluation model.

[0432] Secondly, transmission paths are dynamically prioritized based on the evaluation results, with low-latency and high-availability paths being selected first. For example, if the network latency of a certain path exceeds a threshold, the system automatically switches to a backup path.

[0433] In addition, the algorithm supports differentiated scheduling strategies, allocating exclusive network channels for high-priority data to ensure real-time synchronization of critical messages.

[0434] The intelligent scheduling algorithm is designed as follows:

[0435] Step 1: Assess the health of candidate transmission paths.

[0436] The formula for the health score of each candidate transmission path can be referred to the aforementioned formula (1), and will not be repeated here.

[0437] Step 2: Dynamic weight adjustment.

[0438] Adjusting weights based on real-time traffic prediction results, for example in the following scenarios:

[0439] Increase latency weight during peak traffic periods. Reduce bandwidth weight from 0.4 to 0.5. ; and when the network is congested, increase the load weight , triggering path switching.

[0440] Step 3: Path selection.

[0441] Primary path selection: select the candidate path with the highest health score as the primary transmission path.

[0442] Backup path: select the path with the second highest score as the backup, and automatically switch when the Round Trip Time (RTT) of the primary path (also known as network latency) > 50ms (i.e. the preset network latency) or the packet loss rate > 5% (i.e. the preset packet loss rate).

[0443] Step 4: Feedback optimization.

[0444] Short-term feedback: update the health score of the target path and the remaining candidate paths every 5 minutes, and record the actual transmission delay and prediction deviation.

[0445] Long-term learning: use reinforcement learning to optimize weights , and to minimize the historical average delay.

[0446] S303E2, data compression and batch processing.

[0447] To reduce network transmission overhead, adaptive data compression technology is also introduced to match the corresponding target compression algorithm for vehicle message data of different importance levels.

[0448] For low-priority vehicle message data, a high-compression-rate algorithm (e.g. Zstantdrd compression algorithm) is used to reduce transmission volume; for high-priority vehicle message data, a low-latency compression algorithm (e.g. Snappy compression algorithm) is selected to balance efficiency and real-time performance.

[0449] At the same time, intelligent batch processing mechanism can also be implemented to combine small messages into batch messages for transmission. The batch processing window size is dynamically adjusted according to the predicted traffic: the window size is reduced during traffic peak to reduce delay, and the window size is expanded during traffic trough to increase throughput.

[0450] S303E3, fault recovery and consistency guarantee.

[0451] The module designs a multi-level fault recovery mechanism to deal with network interruptions or cluster abnormalities. When synchronization failure is detected, the system first attempts automatic retry, and gradually increases the retry interval to avoid the snowball effect. If the retry fails, the local cache mechanism is triggered to temporarily store data to the disk queue, and then re-synchronize after the network is restored.

[0452] To ensure data consistency, the module implements Exactly-Once semantics based on the Kafka transaction mechanism, and prevents data loss or duplication through regular offset checks. In addition, the system records synchronization checkpoints, supports fast recovery from breakpoints, and reduces resource consumption caused by full synchronization.

[0453] S303E4, load balancing technology.

[0454] Load balancing technology avoids single node overload by dynamically allocating synchronization tasks. The module deploys a distributed load balancer between cloud and on-premise clusters, and monitors the message accumulation and processing delay of each Broker in real time. Based on the monitoring data, the synchronization task is allocated to the node with lighter load using the weighted round robin algorithm, and the high-load node is protected by flow limiting. To solve the problem of uneven load at partition level, the module periodically analyzes the message distribution of each partition, and automatically adjusts the mapping relationship between partitions and Brokers using the partition redistribution tool provided by Kafka, to maximize resource utilization.

[0455] It should be noted that although the steps of the method in the present application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired result. In addition or alternatively, some steps can be omitted, a plurality of steps can be combined into one step, and / or one step can be divided into a plurality of steps, etc.; or, steps in different embodiments can be combined into a new technical solution.

[0456] Based on the same inventive concept as the foregoing embodiments, the embodiments of the present application provide a cloud resource processing apparatus and a prediction model training apparatus, both of which include respective modules and units included in the respective modules, and can be implemented by a processor; of course, they can also be implemented by a specific logic circuit; in the implementation process, the processor can be an AI acceleration engine (such as NPU, etc.), GPU, central processing unit (CPU), microprocessor (MPU), digital signal processor (DSP), or field programmable gate array (FPGA), etc.

[0457] Figure 4 A structural schematic diagram of a cloud resource processing apparatus provided by an embodiment of the present application is shown in Figure 4 The cloud resource processing apparatus 40 includes a first determination unit 401 and a first adjustment unit 402.

[0458] The first determination unit 401 is configured to determine time features, regional features, and vehicle type features based on time information, regional information, and vehicle type information to be processed by the cloud resource.

[0459] The first adjusting unit 402 is configured to process the time feature, the region feature and the vehicle type feature through the prediction model, predict the vehicle message flow to be input to the cloud device, and obtain a prediction result; and adjust the configuration of the cloud resource based on the prediction result, so that the configuration of the cloud resource matches the prediction result.

[0460] The prediction result is used to represent the size of the current vehicle message flow to be input to the cloud device.

[0461] In some embodiments, the first adjusting unit 402 is further configured to: in a case where the current vehicle message flow to be input to the cloud device represented by the prediction result exceeds a first processing capability of the cloud resource, increase a throughput node in the cloud resource; and in a case where the current vehicle message flow to be input to the cloud device represented by the prediction result is lower than a second processing capability of the cloud resource, reduce the throughput node in the cloud resource.

[0462] The first processing capability is higher than the second processing capability.

[0463] In some embodiments, the first adjusting unit 402 is further configured to: in a case where the current vehicle message flow to be input to the cloud device represented by the prediction result is greater than a second threshold value, increase a consumer instance in the cloud resource; and in a case where the current vehicle message flow to be input to the cloud device represented by the prediction result satisfies a first preset condition, reduce the consumer instance in the cloud resource.

[0464] The first preset condition includes that the current vehicle message flow to be input to the cloud device represented by the prediction result is less than a third threshold value, or the current vehicle message flow to be input to the cloud device represented by the prediction result is less than the third threshold value and exceeds a specified time length.

[0465] The more the number of consumer instances, the faster the capability of the cloud resource to process the vehicle message flow; and the less the number of consumer instances, the slower the capability of the cloud resource to process the vehicle message flow.

[0466] The third threshold value is less than or equal to the second threshold value.

[0467] In some embodiments, the first adjusting unit 402 is further configured to: in a case where a first vehicle message flow in the current vehicle message flow to be input to the cloud device represented by the prediction result exceeds a first threshold value, determine an increase amount of the first vehicle message flow; and increase a partition in a first region in the cloud resource corresponding to processing the first vehicle message flow according to the increase amount of the first vehicle message flow, so that the first region after the increase of the partition can process the first vehicle message flow.

[0468] In some embodiments, the first adjusting unit 402 is further configured to: based on the current vehicle message flow to be input to the cloud device represented by the prediction result, determine that the current vehicle message flow belongs to a trough period of the vehicle message flow; based on priorities of vehicle message flows being processed in the cloud resources, determine at least two low-activity areas in the cloud resources; determine a target area and an idle area in the at least two low-activity areas; the target area is used to process the current vehicle message flow and vehicle message flows being processed in the two low-activity areas; and the idle area is used as a backup area.

[0469] In some embodiments, the first adjusting unit 402 is further configured to: divide each vehicle message in the current vehicle message flow to be input to the cloud device represented by the prediction result according to importance of vehicle message traffic, to determine an importance level of each vehicle message; determine that an area processing high-importance-level vehicle messages is an independent area in the cloud resources, or determine that a throughput node processing high-importance-level vehicle messages is an independent throughput node; the high-importance-level vehicle message is a vehicle message including fault alarm information; and low-importance-level vehicle messages are compressed before being transmitted; the low-importance-level vehicle message is a vehicle message including regular state information.

[0470] In some embodiments, the cloud resource processing apparatus provided by the embodiments of the present application further includes a first processing unit 403 (not shown), a second determining unit 404 (not shown), and a first compression and transmission unit 405 (not shown).

[0471] The first processing unit 403 is configured to: based on the cloud resources, process the current vehicle message flow to be input to the cloud device represented by the prediction result, to obtain vehicle message flow to be sent.

[0472] The second determining unit 404 is configured to: based on importance levels of each vehicle message in the vehicle message flow to be sent, determine target compression algorithms corresponding to vehicle messages of different importance levels.

[0473] The first compression and transmission unit 405 is configured to: each vehicle message of different importance levels is transmitted after being compressed by a corresponding target compression algorithm.

[0474] For high-importance-level vehicle messages in the vehicle message flow to be sent, the target compression algorithm is a low-delay compression algorithm; the high-importance-level vehicle message is a vehicle message including fault alarm information; for low-importance-level vehicle messages in the vehicle message flow to be sent, the target compression algorithm is a high-compression-rate algorithm; and the low-importance-level vehicle message is a vehicle message including regular state information.

[0475] In some embodiments, the first compression transmission unit 405 is further configured to determine health scores of at least two candidate transmission paths of the vehicle message traffic to be sent; select, from the at least two candidate transmission paths, a transmission path with the highest health score as a target transmission path; and transmit the compressed vehicle messages of different importance levels by using the target transmission path.

[0476] The health of the transmission path is used to evaluate network delay, network bandwidth, and network load of the transmission path.

[0477] In some embodiments, the first compression transmission unit 405 is further configured to, for each of the at least two candidate transmission paths, perform: determining network delay, network bandwidth, and network load of the candidate transmission path; and determining a health score of the candidate transmission path according to the network delay, the network bandwidth, and the network load of the candidate transmission path.

[0478] In some embodiments, the cloud resource processing apparatus provided by the embodiments of the present application further includes a first switching unit 406 (not shown).

[0479] The first switching unit 406 is configured to, in the case of an abnormality of the target transmission path, switch to a backup transmission path to transmit the compressed vehicle messages of different importance levels by using the backup transmission path.

[0480] The backup target transmission path is a candidate transmission path with the second highest health score.

[0481] Figure 5 A prediction model training apparatus provided by an embodiment of the present application, as shown in Figure 5 The prediction model training apparatus 50 includes a third determination unit 501 and a first training unit 502.

[0482] The third determination unit 501 is configured to determine time features, regional features, and vehicle type features based on time information, regional information, and vehicle type information of historical vehicle message traffic to form a first feature data set; vehicle message traffic of different vehicle types is different; vehicle message traffic of different regions is different; and vehicle message traffic of different times is different.

[0483] The first training unit 502 is configured to train a candidate model by using the first feature data set to obtain a prediction model; and dynamically adjust configuration of a cloud device according to a prediction result of the prediction model; the prediction result is used to represent a size of current vehicle message traffic to be input to the cloud device.

[0484] In some embodiments, when the candidate model comprises a plurality, the first training unit 502 is further configured to: filter the first feature dataset to obtain a second feature dataset by using a correlation analysis algorithm and / or a feature importance evaluation algorithm; train each candidate model in the plurality of candidate models by using the second feature dataset; evaluate the prediction accuracy and real-time performance of each trained candidate model, and select a candidate model that meets a preset evaluation condition as the prediction model.

[0485] Alternatively, the first training unit 502 is further configured to: evaluate the prediction accuracy and real-time performance of each trained candidate model, and determine the prediction model according to the trained candidate models and the weights corresponding to the candidate models; the weights corresponding to the candidate models are related to the prediction accuracy and real-time performance of the candidate models.

[0486] The above description of the device embodiments is similar to the description of the method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0487] It should be noted that the division of modules in the embodiments of the present application is illustrative, and is only a logical function division. In actual implementation, another division mode can be used. In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or can be physically separated, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware, or in the form of a software functional unit. It can also be realized in the form of a combination of software and hardware.

[0488] It should be noted that in the embodiments of the present application, if the above method is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing an electronic device to execute all or part of the methods described in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and various storage media that can store program codes. Thus, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0489] The embodiments of the present application provide an electronic device, Figure 6 The structure of the electronic device provided in the embodiments of the present application is shown in the figure Figure 6As shown, the electronic device 60 includes a memory 601 and a processor 602, the memory 601 stores a computer program executable on the processor 602, and the processor 602 implements the steps in the method provided in the above embodiments when executing the program.

[0490] It should be noted that the memory 601 is configured to store instructions and applications executable by the processor 602, and can also buffer data (for example, image data, audio data, voice communication data and video communication data) to be processed or having been processed by the processor 602 and each module in the electronic device 60, which can be implemented by FLASH or Random Access Memory (RAM).

[0491] The embodiments of the present application further provide a computer readable storage medium for storing the computer program.

[0492] Optionally, the computer readable storage medium can be applied to the electronic device in the embodiments of the present application, and the computer program makes the processor or the electronic device execute the methods of the embodiments of the present application, which will not be described herein for the sake of brevity.

[0493] The embodiments of the present application further provide a computer program product including computer program instructions.

[0494] Optionally, the computer program product can be applied to the electronic device in the embodiments of the present application, and the computer program instructions make the processor or the electronic device execute the methods of the embodiments of the present application, which will not be described herein for the sake of brevity.

[0495] The embodiments of the present application further provide a computer program.

[0496] Optionally, the computer program can be applied to the electronic device in the embodiments of the present application, and when the computer program runs on the processor or the electronic device, makes the processor or the electronic device execute the methods of the embodiments of the present application, which will not be described herein for the sake of brevity.

[0497] It should be noted that the above descriptions of the electronic device, the storage medium, the computer program product and the computer program embodiments are similar to the descriptions of the method embodiments, and have similar beneficial effects. For technical details not disclosed in the electronic device, the storage medium, the computer program product and the computer program embodiments of the present application, please refer to the description of the method embodiments.

[0498] It should be understood that the term "one embodiment" or "an embodiment" or "some embodiments" as used herein means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. Therefore, the appearances of the phrase "in one embodiment" or "in an embodiment" or "in some embodiments" in various places throughout the specification are not necessarily referring to the same embodiment. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that the sequence of steps in the above-described processes does not necessarily mean that the steps are executed in the order described, and the execution order of the steps should be determined according to their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the application. The sequence numbers of the above-described embodiments of the application are only for description, and do not represent the advantages or disadvantages of the embodiments. The above description of each embodiment tends to emphasize the differences between the embodiments, and the same or similar parts can be mutually referred to, and are not described herein for the sake of brevity.

[0499] The term "and / or", as used herein, merely describes association between associated objects, and can mean that three cases exist, for example, object A and / or object B can mean that object A exists alone, object A and object B exist together, and object B exists alone.

[0500] It should be noted that the terms "comprising", "including", or any other variant are intended to cover a non-exclusive inclusion, such that processes, methods, articles, or devices that comprise a list of elements not only include those elements, but also include other elements not expressly listed or inherent to such processes, methods, articles, or devices. Without more limitations, an element defined by the statement "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or device including the element.

[0501] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The above-described embodiments are merely illustrative, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed components can be indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0502] The modules described above as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules; they can be located in one place or distributed on multiple network units; and some or all of the modules can be selected as needed to achieve the purposes of the embodiments.

[0503] In addition, the functional modules in the embodiments of the present application can be integrated in one processing unit, or each module can be a separate unit, or two or more modules can be integrated in one unit; the integrated modules can be realized in the form of hardware or in the form of hardware plus software functional units.

[0504] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware, and the foregoing program can be stored in a computer readable storage medium, and the program executes the steps including the above-mentioned method embodiments when executed; and the foregoing storage medium includes mobile storage devices, read only memory (ROM), magnetic discs or optical discs and various storage medium that can store program codes.

[0505] Alternatively, the integrated units of the present application, if implemented in the form of software functional modules and sold or used as independent products, can also be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software products, and the computer software products are stored in a storage medium, including a number of instructions to make an electronic device execute all or part of the methods described in the embodiments of the present application. The foregoing storage medium includes mobile storage devices, ROM, magnetic discs or optical discs and various storage medium that can store program codes.

[0506] The methods disclosed in the several method embodiments provided by the present application can be combined arbitrarily without conflict to obtain new method embodiments.

[0507] The features disclosed in the several product embodiments provided by the present application can be combined arbitrarily without conflict to obtain new product embodiments.

[0508] The features disclosed in the several method or device embodiments provided by the present application can be combined arbitrarily without conflict to obtain new method embodiments or device embodiments.

[0509] The above merely provides the implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of the change or replacement within the technical range disclosed by the present application, which should be covered in the protection scope of the present application.

Claims

1. A processing method of cloud resources, characterized by, The method is applied to a cloud device, and the method comprises: determining time characteristics, region characteristics and vehicle type characteristics based on time information, region information and vehicle type information of the cloud resources to be processed; processing the time characteristics, the region characteristics and the vehicle type characteristics by a prediction model to predict vehicle message flow to be input to the cloud device and obtain a prediction result; the prediction result is used to represent the size of the current vehicle message flow to be input to the cloud device; adjusting the configuration of the cloud resources based on the prediction result to match the configuration of the cloud resources with the prediction result; wherein the adjusting the configuration of the cloud resources based on the prediction result comprises: when the current vehicle message flow to be input to the cloud device represented by the prediction result belongs to a trough period of vehicle message flow: determining at least two low-activity regions in the cloud resources based on the priority of the vehicle message flow being processed in the cloud resources; determining a target region and an idle region in the at least two low-activity regions; the target region is used to process the current vehicle message flow and the vehicle message flow being processed in the at least two low-activity regions; and the idle region is used as a backup region.

2. The method of claim 1, wherein, the adjusting the configuration of the cloud resources based on the prediction result comprises: when the current vehicle message flow to be input to the cloud device represented by the prediction result exceeds a first processing capacity of the cloud resources, adding a throughput node in the cloud resources; when the current vehicle message flow to be input to the cloud device represented by the prediction result is lower than a second processing capacity of the cloud resources, reducing a throughput node in the cloud resources; the first processing capacity is higher than the second processing capacity.

3. The method of claim 1, wherein, the adjusting the configuration of the cloud resources based on the prediction result comprises: when a first vehicle message flow in the current vehicle message flow to be input to the cloud device represented by the prediction result exceeds a first threshold value, determining an increase amount of the first vehicle message flow; increasing a partition in a first region in the cloud resources corresponding to processing the first vehicle message flow according to the increase amount of the first vehicle message flow, so that the first region after increasing the partition can process the first vehicle message flow.

4. The method of claim 1, wherein, the adjusting the configuration of the cloud resources based on the prediction result comprises: dividing each vehicle message in the current vehicle message flow to be input to the cloud device represented by the prediction result according to the importance of vehicle message services to determine the importance level of the each vehicle message; determining a region processing high-importance-level vehicle messages as an independent region in the cloud resources, or determining a throughput node processing the high-importance-level vehicle messages as an independent throughput node; the high-importance-level vehicle message is a vehicle message including fault alarm information; compressing low-importance-level vehicle messages before transmission; the low-importance-level vehicle message is a vehicle message including regular state information.

5. The method according to any one of claims 1 to 4, characterized in that, After the configuration of the cloud resource is adjusted based on the prediction result to match the prediction result, the method further includes: processing the prediction result based on the cloud resource to process current vehicle message traffic to be input to the cloud device to obtain vehicle message traffic to be sent out; determining target compression algorithms corresponding to vehicle messages of different importance levels based on importance levels of vehicle messages in the vehicle message traffic to be sent out; transmitting vehicle messages of different importance levels after compression by corresponding target compression algorithms; wherein, for vehicle messages of high importance levels in the vehicle message traffic to be sent out, the target compression algorithm is a low-delay compression algorithm; the vehicle messages of high importance levels are vehicle messages including fault alarm information; for vehicle messages of low importance levels in the vehicle message traffic to be sent out, the target compression algorithm is a high-compression-rate algorithm; the vehicle messages of low importance levels are vehicle messages including regular state information.

6. The method of claim 5, wherein, The transmission of vehicle messages of different importance levels after compression by corresponding target compression algorithms includes: determining health scores of at least two candidate transmission paths of the vehicle message traffic to be sent out; selecting, among the at least two candidate transmission paths, a transmission path with the highest health score as a target transmission path; transmitting vehicle messages of different importance levels after compression by using the target transmission path; wherein, the health of the transmission path is used to evaluate network delay, network bandwidth, and network load of the transmission path.

7. The method of claim 6, wherein, The determination of the health scores of the at least two candidate transmission paths of the vehicle message traffic to be sent out includes: for each of the at least two candidate transmission paths, performing: determining network delay, network bandwidth, and network load of the candidate transmission path; determining the health score of the candidate transmission path according to the network delay, network bandwidth, and network load of the candidate transmission path.

8. The method of claim 6, wherein, The method further includes: in the case of an exception of the target transmission path, switching to a backup transmission path to transmit vehicle messages of different importance levels after compression by using the backup transmission path; the backup transmission path is a candidate transmission path with the second highest health score.

9. A processing device of a cloud resource, characterized by, The device is deployed in a cloud device, and the device includes a first determination unit and a first adjustment unit. The first determination unit is configured to determine time features, regional features, and vehicle type features based on time information, regional information, and vehicle type information to be processed by the cloud resource. The first adjustment unit is configured to process the time features, the regional features, and the vehicle type features by a prediction model to predict vehicle message traffic to be input to the cloud device to obtain a prediction result; and adjust the configuration of the cloud resource based on the prediction result to match the prediction result. The prediction result is used to represent the size of current vehicle message traffic to be input to the cloud device. The first adjusting unit is further configured to determine, based on the predicted result representing the current vehicle message flow to be input to the cloud device, that the current vehicle message flow belongs to a low-peak period of vehicle message flow, and determine at least two low-activity areas in the cloud resource based on the priority of the vehicle message flow being processed in the cloud resource; determine a target area and an idle area in the at least two low-activity areas; the target area is used to process the current vehicle message flow and the vehicle message flow being processed in the at least two low-activity areas; and the idle area is used as a backup area.

10. An electronic device, comprising: The electronic device comprises a memory and a processor, and the memory stores a computer program or instructions; when the computer program or the instructions are executed by the processor, the method in any one of claims 1-8 is implemented.

Images