Data acquisition optimization method, system, device, equipment and program product

Abstract

The invention provides a data acquisition optimization method, system, device, equipment and program product, and relates to the technical field of intelligent driving. The method comprises the steps that a data collection log is obtained, and the data collection log comprises vehicle intelligent driving data collected based on a data collection demand; determining the data subscription of each log type according to the data acquisition log; obtaining a data subscription strategy optimized according to the data subscription amount of each log type; and adjusting the data acquisition demand based on the optimized data subscription strategy. The problem that in the prior art, intelligent driving data collection efficiency and quality are insufficient is solved.

Description

Technical Field

[0001] The present disclosure relates to the field of intelligent driving technology, and in particular to a data acquisition optimization method, system, device, equipment and program product. Background Art

[0002] At present, mass-produced vehicles collect data based on the data collection requirements of the in-vehicle configuration and upload it to the cloud for storage. When in use, engineers obtain the corresponding data from the cloud. The engineers' demand for cloud data cannot be directly fed back to the data collection process, and the data collection efficiency and quality are insufficient. Summary of the invention

[0003] In view of this, the embodiments of the present disclosure provide a data collection optimization method, system, device, equipment and program product to solve the problems of insufficient efficiency and quality of intelligent driving data collection in the prior art.

[0004] In a first aspect, the present disclosure provides a data collection optimization method, comprising:

[0005] Acquire a data collection log, wherein the data collection log includes intelligent driving data of the vehicle collected based on data collection requirements;

[0006] Determining a data subscription amount for each log type according to the data collection log;

[0007] Obtaining a data subscription strategy optimized according to the data subscription amount of each of the log types;

[0008] The data collection requirement is adjusted based on the optimized data subscription strategy.

[0009] In a second aspect, the present disclosure provides a data collection optimization system, including a vehicle-side device and a cloud device;

[0010] The vehicle-side device is used to collect the intelligent driving data of the vehicle based on data collection requirements and upload it to the cloud device;

[0011] The cloud device is used to obtain a data collection log, wherein the data collection log includes intelligent driving data of the vehicle collected based on data collection requirements; determine the data subscription amount of each log type according to the data collection log; obtain a data subscription strategy optimized according to the data subscription amount of each log type; and adjust the data collection requirements based on the optimized data subscription strategy.

[0012] In a third aspect, the present disclosure provides a data collection optimization device, comprising:

[0013] An acquisition module, used to acquire a data acquisition log, wherein the data acquisition log includes intelligent driving data of the vehicle collected based on data acquisition requirements;

[0014] A determination module, used to determine the data subscription amount of each log type according to the data collection log;

[0015] An optimization module, used to obtain a data subscription strategy optimized according to the data subscription amount of each log type;

[0016] An adjustment module is used to adjust the data collection demand based on the optimized data subscription strategy.

[0017] In a fourth aspect, the present disclosure provides an electronic device, including:

[0018] at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores at least one computer program executable by the at least one processor, and the at least one computer program is executed by the at least one processor so that the at least one processor can perform the data acquisition optimization method as described in the first aspect.

[0021] In a fifth aspect, the present disclosure provides a computer program product, which includes a computer program. When the computer program is executed in a processor, the data acquisition optimization method described in the first aspect is implemented.

[0022] The embodiment provided by the present disclosure obtains a data collection log, wherein the data collection log includes intelligent driving data of the vehicle collected based on data collection requirements; determines the data subscription amount of each log type according to the data collection log; obtains a data subscription strategy optimized according to the data subscription amount of each log type; and adjusts the data collection requirements based on the optimized data subscription strategy. This realizes automatic analysis of the data collection log, provides a basis for optimizing the data subscription strategy by counting the data subscription amount of each log type, and then adjusts the data collection requirements of the vehicle-side device based on the optimized data subscription, thereby improving data collection efficiency and quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0024] Figure 1 The figure is a schematic diagram of the data acquisition optimization method flow in the embodiment of the present disclosure;

[0025] Figure 2 The figure shows a schematic diagram of the framework of the data acquisition system in the embodiment of the present disclosure;

[0026] Figure 3 The figure shows a schematic diagram of the system architecture of the Kafka log collection module in an embodiment of the present disclosure;

[0027] Figure 4 The figure is a schematic diagram of the Topic log processing process in the embodiment of the present disclosure;

[0028] Figure 5 The figure is a schematic diagram of the structure of the data acquisition optimization device in the embodiment of the present disclosure;

[0029] Figure 6 FIG. 1 is a schematic diagram of the structure of an electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0031] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0032] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0033] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "including" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. "Connected" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0034] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0035] Overview

[0036] Traditional autonomous driving systems rely on multiple independent modules, such as perception, planning, and control, each driven by a specific algorithm. However, this approach faces challenges in handling complex driving scenarios because the interfaces and data conversion between modules may lead to information loss and incoherent decisions.

[0037] The introduction of the end-to-end big model aims to achieve the integration of perception and decision-making through a single training process of deep neural networks directly from raw sensor data to driving decisions. This integrated approach can use the powerful learning ability of the big model to automatically learn complex driving strategies from massive data, thereby improving the adaptability and robustness of the autonomous driving system. For example, a manufacturer applied the end-to-end big model to the FSD (Full Self-Driving) Version 12 of smart cars; a manufacturer launched an end-to-end big model for mass production: neural network XNet + regulatory control big model XPlanner + big language model XBrain, demonstrating the innovative application of this technology in the automotive intelligent driving industry.

[0038] The closed loop of mass-produced vehicles plays a vital role in the end-to-end big model. Compared with the traditional data processing process, the end-to-end big model emphasizes automatic data mining, labeling, training and simulation testing, which not only accelerates the iteration speed of the model, but also optimizes the autonomous driving system's understanding and response capabilities to the environment through deep learning.

[0039] As technology continues to mature, end-to-end big models are guiding intelligent driving towards a higher level of automation. Through the integrated layout of cloud, pipe and end, it aims to achieve a comprehensive autonomous driving solution, which not only meets the most essential pursuit of intelligent technology, but also is a powerful breakthrough in the "ceiling" of existing technology. In this context, the intelligent driving data collection system based on end-to-end big models aims to optimize data collection efficiency and quality, provide richer and more accurate data support for model training, and promote the rapid iteration and safe application of intelligent driving technology.

[0040] Based on this, a data collection optimization method is proposed for the intelligent driving data collection system based on the end-to-end large model to improve the data collection efficiency and quality.

[0041] Exemplary Methods

[0042] The present disclosure provides a data collection optimization method, which is applied to a cloud device, such as Figure 1 As shown, the method mainly includes the following steps:

[0043] Step 101, obtaining a data collection log, wherein the data collection log includes intelligent driving data of the vehicle collected based on data collection requirements.

[0044] In some embodiments, the cloud device receives the intelligent driving data collected by the vehicle-side device and generates a data collection log, wherein the data collection log includes the intelligent driving data based on a preset log format.

[0045] The preset log format is pre-configured in the cloud device, and the intelligent driving data collected by the vehicle-side device is formatted in a unified manner according to the preset log format.

[0046] Step 102: Determine the data subscription amount for each log type according to the data collection log.

[0047] In some embodiments, the data subscription amount of each log type is determined based on the data collection log, including: for each of the data collection logs, determining the similarity between the data collection log and a log template of each log type, and determining the log type to which the data collection log belongs based on the similarity corresponding to each log type; for each of the log types, counting the number of data collection logs to which the log type belongs, and determining the data subscription amount of the log type based on the counted number.

[0048] In some embodiments, determining the similarity between the data collection log and the log template of each log type includes:

[0049] For each log type, the log template is processed as follows:

[0050] Compare the i-th marked data segment in the log template with the i-th marked data segment in the data collection log set, and generate a similarity representation value corresponding to the i-th marked data segment according to the comparison result; i is an integer greater than or equal to 1 and less than or equal to N, and N is the total number of marked data segments included in the log template;

[0051] The similarity between the data collection log and the log template is determined according to the similarity representation value corresponding to each of the marked data segments in the log template.

[0052] In some embodiments, generating the similarity representation value corresponding to the i-th marked data segment based on the comparison result includes: when the comparison result is equal, determining the similarity representation value corresponding to the i-th marked data segment to be a first preset value; when the comparison result is unequal, determining the similarity representation value corresponding to the i-th marked data segment to be a second preset value.

[0053] In some embodiments, determining the similarity between the data collection log and the log template based on the similarity representation value corresponding to each of the marked data segments in the log template includes: determining the sum of the similarity representation values ​​corresponding to each of the marked data segments in the log template; and taking the ratio of the obtained sum to N as the similarity between the data collection log and the log template.

[0054] Specifically, the calculation formula of similarity is expressed as:

[0055]

[0056] Wherein, Correlation represents similarity, Equation represents the similarity representation value, Reference(i) represents the i-th marked data segment of the log template, and Number(i) represents the i-th marked data segment of the data collection log. In this formula, it is assumed that the first preset value is 1 and the second preset value is 0. The first preset value and the second preset value can be set as needed, which is only an example here.

[0057] For example, assuming that a log template is: "[LogType]: VehicleCommandData Topic issubscribed with info[InfoData]"; a data collection log is: "[Info]: VehicleCommandDataTopic is subscribed with info[gear]", the similarity representation values ​​of each marked data segment are shown in Table 1:

[0058] Table 1 Similarity representation values ​​of labeled data segments

[0059]

[0060]

[0061] The similarity between the data collection log and the log template is expressed as:

[0062]

[0063] This means that the data collection log in this example has a 71% similarity with the log template.

[0064] In some embodiments, determining the log type to which the data collection log belongs based on the similarity corresponding to each of the log types includes: for each of the log types, when the similarity corresponding to the log template exceeds a similarity threshold corresponding to the log type, determining the log type as the log type to which the data collection log belongs.

[0065] The same similarity threshold may be pre-configured for each log type; or a corresponding similarity threshold may be configured for each log type, and the similarity thresholds corresponding to two log types may be the same or different.

[0066] For example, assuming that the similarity threshold corresponding to the log type is 0.6, and the similarity between the data collection log and the log template of the log type is 0.71, then 0.71>0.6, and the data collection log belongs to the log type.

[0067] Step 103: Obtain a data subscription strategy optimized according to the data subscription amount of each log type.

[0068] Exemplarily, the data subscription strategy optimized according to the data subscription volume of each of the log types includes: when the data subscription volume of the log type is less than the subscription volume threshold, the optimized data subscription strategy no longer subscribes to the data of the log type; when the data volume of the log type is not less than the subscription volume threshold, the optimized data subscription strategy continues to subscribe to the data of the log type.

[0069] For example, the number of data topics (Topics) for the two log types of high-speed navigation (NOA) and automatic emergency braking (AEB) is large, while the number of data topics (Topics) for the two log types of low-speed exploration (LSDA) and reverse tracking (TBA) is small. Therefore, by adjusting the data subscription strategy, it is automatically adjusted to continue collecting data topics with large subscriptions and stop collecting data topics with small subscriptions.

[0070] This is just an example. Depending on the subscription volume of each log type and the different application scenarios, the optimization method for the data subscription strategy is also different. The specific optimization method can be configured as needed.

[0071] Exemplarily, a data subscription strategy optimized according to the data subscription quantity of each log type is obtained, including: the cloud device transmits the data subscription quantity of each log type to the technician's terminal interface for display; the technician adjusts the data subscription strategy according to the data subscription quantity of each log type and his own needs and configures it to the cloud device; the cloud device adjusts the data collection requirements according to the adjusted data subscription strategy and configures it to the vehicle-side device.

[0072] This is just an example. Manual optimization of the data subscription policy by a technician is just an example. The data subscription policy can also be automatically optimized through pre-configured optimization rules. The optimization rules can be configured by the technician according to the needs.

[0073] Step 104: Adjust the data collection requirements based on the optimized data subscription strategy.

[0074] Exemplarily, the data collection requirements are adjusted based on the optimized data subscription policy. For log types whose data subscription policy is to stop subscription, the data collection requirements are adjusted to no longer collect data; for log types whose data subscription policy is to continue subscription, the data collection requirements are adjusted to continue collecting data.

[0075] The adjusted data collection requirements are configured to the vehicle-side equipment, so that the vehicle-side equipment installs the adjusted data collection requirements to perform intelligent driving data collection.

[0076] The embodiment provided by the present disclosure obtains a data collection log, wherein the data collection log includes intelligent driving data of the vehicle collected based on data collection requirements; determines the data subscription amount of each log type based on the data collection log; obtains a data subscription strategy optimized according to the data subscription amount of each log type; and adjusts the data collection requirements based on the optimized data subscription strategy. This realizes automatic analysis of the data collection log, provides a basis for optimizing the data subscription strategy by counting the data subscription amount of each log type, and then adjusts the data collection requirements of the vehicle-side device based on the optimized data subscription, establishes a relationship between data collection and use, and realizes the optimization of data collection based on data use.

[0077] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation, the specific execution order of each step should be determined by its function and possible internal logic, and the execution order between the steps is not limited to being implemented according to the step number.

[0078] Exemplary Systems

[0079] The present disclosure also provides a data collection optimization system, including a vehicle-side device and a cloud device;

[0080] The vehicle-side device is used to collect the intelligent driving data of the vehicle based on data collection requirements and upload it to the cloud device;

[0081] The cloud device is used to obtain a data collection log, wherein the data collection log includes intelligent driving data of the vehicle collected based on data collection requirements; determine the data subscription amount of each log type according to the data collection log; obtain a data subscription strategy optimized according to the data subscription amount of each log type; and adjust the data collection requirements based on the optimized data subscription strategy.

[0082] Exemplarily, the data acquisition optimization system provided by the embodiment of the present disclosure is based on a data acquisition system, such as Figure 2 The figure shows the framework of the data acquisition system. The vehicle-side equipment includes three modules: domain control hardware, vehicle-mounted domain control operating system, and vehicle-side data closed-loop engine; the cloud-side equipment includes four modules: Infrastructure as a Service (IaaS), Platform as a Service (PaaS), application service layer, and Software as a Service (SaaS) ecological layer. These seven modules constitute a data acquisition optimization system based on an end-to-end large model.

[0083] Module 1

[0084] The domain control hardware is called Autonomous Driving Control Unit (ADCU), which is a domain controller designed for intelligent driving functions above Level 2. It is designed and developed based on System on Chip (SOC) and Microcontroller Unit (MCU), where SOC is used for complex computing tasks such as environmental perception, image fusion, and path planning; while MCU is responsible for functions with high real-time and safety requirements such as safety monitoring, redundant control, gateway communication, and vehicle control.

[0085] Data collection and processing: In intelligent driving scenarios, ADCU collects environmental data through a variety of sensors (such as cameras, radars, lidars, etc.), which are transmitted to the SOC for processing. The dedicated processor in the SOC pre-processes, extracts features, and fuses the raw sensor data to form a comprehensive understanding of the surrounding environment. Based on this information, ADCU uses complex algorithms to make decisions, such as path planning, obstacle avoidance strategies, etc.

[0086] ADCU not only performs key intelligent driving functions on the vehicle, but also has the ability to transmit sensor data and vehicle status information back to the cloud. This feature is crucial for the continuous optimization of autonomous driving algorithms:

[0087] Data backhaul: ADCU collects data from sensors such as cameras, radars, lidars, and vehicle operating status information, and uploads this data to the cloud through V2X (Vehicle to Everything) communication technology or other wireless networks.

[0088] Algorithm optimization: The massive amount of data received in the cloud can be used to train and verify the autonomous driving algorithm model. Through in-depth learning of actual driving scenarios, the algorithm can be continuously iterated and upgraded to improve the accuracy and robustness of decision-making.

[0089] OTA upgrade: Based on the results of cloud data analysis, vehicle manufacturers can implement OTA (Over-the-Air) software updates to push new features to users or improve existing features to ensure the continuous evolution and safety performance of intelligent driving systems.

[0090] This closed-loop data flow mechanism is the key to modern smart cars realizing full-scenario travel solutions. It not only enhances the vehicle's autonomous driving capabilities, but also provides users with a more personalized and safe driving experience.

[0091] Module 2

[0092] The operating system is the vehicle-mounted domain control operating system in ADCU, which is a software platform specially designed to support high-level autonomous driving functions. It plays a core role in ADCU, responsible for managing and coordinating various computing resources, and processing data from different sensors. The vehicle-mounted domain control operating system is mainly aimed at the field of intelligent driving, supporting functions such as perception, positioning, planning, and decision-making, and has extremely high requirements for safety and reliability. The vehicle-mounted domain control operating system constitutes the core part of intelligent vehicle technology, ensuring the efficient operation of the vehicle and the safety of passengers.

[0093] For data subscription on the vehicle-mounted domain control operating system, we can refer to the process of the OPC-UA (OpenPlatform Communications Unified Architecture) protocol in the industrial Internet of Things. Although the specific steps directly applied to the automotive field may be different, the basic principles are similar:

[0094] Define a subscription: Define a subscription on the domain controller hardware, specifying the data points or signals you are interested in.

[0095] Configuration parameters: Set subscription parameters, such as sampling interval, trigger conditions, etc.

[0096] Connecting to a server: The domain control hardware acts as a client, connecting to a server (which could be another ECU or other sensor device) that provides data.

[0097] Receive data: The domain control hardware will receive updates as data becomes available. This may include real-time sensor readings, diagnostic information, etc.

[0098] Data processing: The received data can be processed as required, such as stored, displayed or further analyzed.

[0099] Response action: Based on the processed data, the domain control hardware can perform corresponding actions, such as adjusting vehicle settings, issuing warnings, or controlling other system components.

[0100] This mechanism allows the on-board domain control operating system to efficiently monitor and respond to changes in vehicle status, and is one of the key technologies for realizing the functions of modern smart cars.

[0101] Module 3

[0102] The vehicle-side data closed-loop engine is closely dependent on the vehicle operating system, but its essence is more like an integrated software framework rather than just an SDK (Software Development Kit). The vehicle-side data closed-loop engine in ADCU combines multiple functional modules, such as task triggers, data collection, data upload, cloud configuration loading, data desensitization and version management, to achieve continuous optimization of the performance of the autonomous driving system.

[0103] In this process, the vehicle-mounted domain control operating system plays the role of a basic platform, which provides the necessary operating environment and support for the closed-loop engine. For example, the kernel is responsible for managing the system's processes, memory, device drivers, etc., to ensure that the data closed-loop engine can perform tasks stably and efficiently; the middleware is located between the application layer and the operating system, helping to coordinate the communication and data exchange between different components. They provide a rich set of APIs and tools, allowing developers to more easily build and optimize the various modules in the closed-loop engine.

[0104] The vehicle-side data closed-loop engine is the core of the vehicle-cloud collaboration in ADCU, which ensures the continuous optimization and safety improvement of the autonomous driving system. A complete vehicle-side data closed-loop engine usually includes the following key modules:

[0105] Cloud configuration loading module: responsible for downloading the latest algorithm models, map information and other configuration parameters from the cloud to ensure that the vehicle's software and hardware are always up to date.

[0106] Task triggers: Initiate data collection and processing tasks based on preset conditions or real-time needs, for example, automatically triggering data recording in specific driving scenarios.

[0107] Data acquisition module: collects raw data from various sensors (such as cameras, radars, lidars, etc.), as well as vehicle status information and environmental parameters.

[0108] Data desensitization module: Process sensitive information to protect user privacy and security while ensuring data compliance.

[0109] Data upload module: uploads the processed data to the cloud server for subsequent analysis and model training. This process may involve compression, encryption and other operations to improve efficiency and security.

[0110] Version management module: Track and manage software versions to ensure that each update can be properly recorded and traced back, which is crucial for troubleshooting and system maintenance.

[0111] Through the collaborative work of these modules, ADCU can form an efficient data closed loop, continuously learn and adapt to complex and changing road environments, thereby improving the reliability of autonomous driving and user experience.

[0112] Module 4

[0113] The IaaS layer adopts a hybrid cloud architecture that combines the advantages of public and private clouds. The IaaS layer provides on-demand computing, storage, and network resources through the Internet, allowing businesses to expand flexibly without having to invest in and maintain physical servers, data centers, and other hardware facilities.

[0114] Public cloud features

[0115] Elastic scaling: Dynamically adjust resources according to actual needs to support high concurrency or burst traffic.

[0116] Global Coverage: Leverage data centers around the world to provide low latency and high availability services.

[0117] Technological innovation: Rapidly introduce new technologies, such as AI and big data analysis, to accelerate business innovation.

[0118] Private cloud advantages

[0119] Data security: Localize sensitive data to enhance data protection and privacy compliance.

[0120] Customized services: Optimize for specific business needs and provide more personalized infrastructure services.

[0121] Cost Control: Reduce long-term operating costs through internal resource management.

[0122] The IaaS layer under the hybrid cloud architecture combines the advantages of both, enjoying the flexibility and economies of scale of the public cloud while retaining the security and customization capabilities of the private cloud. Enterprises can choose the most appropriate deployment environment based on the characteristics of different workloads to achieve optimal configuration and utilization of resources.

[0123] With the support of virtualization technology, the IaaS layer can realize on-demand configuration of hardware resources and provide highly personalized and efficient infrastructure services. Whether it is computing-intensive tasks or large-scale data storage requirements, hybrid cloud IaaS can efficiently meet them in the form of services.

[0124] In terms of data compliance, hybrid cloud architecture provides greater flexibility and control. Sensitive data on the vehicle side must comply with privacy regulations and cannot be transferred to third-party servers at will. By retaining the private cloud portion, enterprises can process this restricted data locally to ensure that it complies with national data security laws.

[0125] At the same time, the public cloud can be used to process non-sensitive or desensitized data, using its powerful computing power and nationwide coverage to accelerate data analysis and application deployment. This layered strategy not only meets compliance requirements, but also fully utilizes the advantages of the public cloud to improve overall business efficiency.

[0126] In summary, IaaS under the hybrid cloud architecture not only ensures low-cost data storage, but also takes into account the needs of data compliance, providing enterprises with a more flexible, efficient and secure data management solution.

[0127] Module 5

[0128] The PaaS layer is mainly responsible for big data processing and AI training in the cloud. The PaaS layer provides rich and flexible service components to support large-scale mass-produced vehicle data processing and analysis.

[0129] Object storage: used to store unstructured data, such as images and videos, and provides high availability and scalability.

[0130] File system: supports the storage and access of structured and semi-structured data, suitable for frequent read and write operations.

[0131] Big data platform: includes Hadoop, Spark and other technology stacks, which are used for batch processing and real-time stream processing of massive data.

[0132] GPU cluster: accelerates the training process of machine learning and deep learning models and improves data processing efficiency.

[0133] Container orchestration: Such as Kubernetes, which enables automated deployment, scaling, and management of applications.

[0134] DevOps CI / CD: Promote collaboration between software development and operation teams to achieve continuous integration and continuous delivery.

[0135] Middleware: Provides functions such as log collection, monitoring and alarm to ensure stable operation of the system and rapid fault location.

[0136] Kafka: As a message queue and event streaming platform, it supports real-time data pipelines and microservices architecture.

[0137] These components work together to build a comprehensive data processing ecosystem, which can be efficiently supported at the PaaS layer from data collection, storage to analysis and application deployment. It not only provides the convenience of "ready-to-use", but also has powerful data processing capabilities and flexibility, suitable for data processing and AI model training in complex scenarios of intelligent driving. Data cleaning and preprocessing are performed at the PaaS layer, and in-depth analysis is performed using the services provided by the computing and storage platform. The entire process reflects the design concept of the big data middle-end architecture, that is, to achieve effective management and value mining of massive data through Paas layer design.

[0138] Module 6

[0139] The application service layer is responsible for data governance of mass-produced vehicle data, completing tasks such as data aggregation, data cleaning, data management, and data task scheduling.

[0140] The vehicle-cloud collaborative cloud engine is responsible for coordinating the data interaction and processing between the cloud and the vehicle. It can collect a large amount of data from vehicle sensors in real time and perform preliminary processing and screening. The vehicle-cloud collaborative cloud engine supports two-way communication. It can dynamically send collection strategy configurations to the vehicle according to data collection needs, helping data managers collect the most needed data and improve data collection efficiency.

[0141] The data management module is responsible for the data lifecycle management of the application service layer, including data storage, circulation and security protection. This module supports efficient retrieval of structured and unstructured data, improving the efficiency of data use.

[0142] The task scheduling module is the command center of the application service layer. It intelligently arranges the execution order and priority of various tasks according to data business requirements and resource conditions. Whether it is a data analysis task, a model training job or a real-time response request, the task scheduling module can ensure the effective use of resources and the timely completion of tasks. At the same time, it also has load balancing capabilities, which can distribute computing tasks among multiple nodes, improving the response speed and processing power of the overall system.

[0143] In summary, the application service layer realizes the improvement process from raw data to business data through the collaborative work of the car-cloud collaborative cloud engine, data management module and task scheduling module.

[0144] Module 7

[0145] The SaaS ecosystem layer is a platform that integrates end-to-end large-scale models of intelligent driving application services, aiming to provide a full range of data application solutions for the research and development of intelligent driving. The following are the key services included in this layer:

[0146] The perception training service provides a training environment for machine learning and deep learning models, supports automatic feature extraction from raw data, and accelerates the model iteration process.

[0147] User behavior mining uses big data analysis technology to deeply understand user driving behavior, rate the driver's driving behavior, and provide a specific data set of good driving behavior for the end-to-end big model.

[0148] Data operations include intelligent driving data query and data subscription for mass-produced vehicles, as well as visualization of intelligent driving functional indicators, showing data indicators such as user usage rate, takeover rate per 100 kilometers, and takeover reasons.

[0149] Simulation services provide support for end-to-end large model training, system testing, and performance optimization by simulating real-world scenarios. They are particularly suitable for complex system verification and user experience prediction.

[0150] High-precision map compliance applications provide high-precision map data processing and distribution services that comply with laws and regulations to meet the needs of industries such as autonomous driving.

[0151] Multi-source data aggregation service integrates data resources from different channels and formats to form a unified view. It promotes the integration and analysis of heterogeneous data and provides comprehensive and in-depth insights for business decisions.

[0152] These services together form the SaaS ecosystem layer, providing rich data applications for end-to-end large models.

[0153] In summary, these seven modules constitute an efficient intelligent driving data acquisition system based on an end-to-end large model, that is, the present disclosure is a data acquisition optimization system provided by an embodiment.

[0154] The most significant feature of this intelligent driving data collection system is that after desensitizing the sensor data, driving operation behavior data and body data on mass-produced vehicles, they are collected to the data center through dynamic subscription in the cloud for intelligent driving function optimization and user driving behavior analysis. In addition, a subscription data mining algorithm is used in the cloud to screen out data sources with large data subscription volumes through machine learning, and to alert data sources with small data subscription volumes, which are then prompted to the cloud management source for optimizing subscription data strategies. Ultimately, data traffic is optimized from the end-to-end, pipe-to-cloud sides, and the hit rate and usage rate of data subscriptions are improved, making the data collection system based on the end-to-end large model more intelligent: solving the problem of insufficient data richness in the data closed loop in the mass production stage; solving the problem of low data utilization in the data closed loop in the mass production stage; solving the problem of insufficient data collection automation rate in the data closed loop in the mass production stage; and solving the problem of high data processing cost in the data closed loop.

[0155] The PaaS layer is used to collect the data subscription logs in Kafka to generate Topic logs for real-time data mining and adjustment of vehicle-side data collection strategies, such as Figure 3 The following is the system architecture diagram of the Kafka log collection module, which mainly includes:

[0156] Job-data-source refers to the source point that reads data from a Kafka topic. This data can be data transmitted to the cloud via HTTPS. In a stream processing application, this is the first stop for data to flow in. For example, define a SourceFunction to consume messages in a Kafka topic. This source point is responsible for continuously pulling data and converting it into a data format that the stream processing engine can understand.

[0157] Job-manager is a core component in the stream processing framework and is responsible for managing the execution lifecycle of jobs. It receives jobs submitted by users and allocates resources (such as CPU and memory) to each task instance. In Apache Flink, job-manager is responsible for scheduling Task Managers and coordinating the communication between them to ensure that jobs are executed correctly according to the specified topology.

[0158] Job-keeper refers to a service used to monitor and maintain job status. It can track whether a job has completed successfully or failed, and restart failed tasks when necessary. This functionality is very important in achieving fault tolerance and persistence.

[0159] Job-runner refers to the component that actually executes the job logic. This corresponds to the task instances on the Task Manager, which run user-defined functions to process data streams. The job-runner is responsible for executing specific calculation logic and working with other runners to ensure the consistency and integrity of the entire job.

[0160] The overall process is as follows:

[0161] Intelligent driving data input, job-data-source reads data from the Kafka topic; job submission, the user submits a stream processing job containing specific business logic to the job-manager; resource allocation and scheduling, job-manager schedules according to available resources and job requirements; task execution, job-runner (that is, the task instance on TaskManager) starts to execute the job logic; result output, the processed data may be written back to another Kafka topic or other data storage; status monitoring and recovery, job-keeper or similar services monitor the job status and trigger the fault recovery mechanism when necessary.

[0162] For each Topic log generated by the above process, follow Figure 4 The Topic log processing process shown is aggregated to obtain the Topic with large subscription volume. Each parent node reads each data collection log from the root node (i.e. Topic log), determines the log type of each data collection log, and obtains a log group including each log type, which includes each data collection log belonging to the corresponding log type.

[0163] In summary, the data collection optimization method and system provided by the embodiments of the present disclosure realize diversified data collection and fusion, and adopt advanced sensor fusion technology to not only integrate multi-source data such as cameras, lidars, millimeter-wave radars, but also realize flexible subscription and fusion of data, significantly improving the richness and diversity of data. For example, by collecting driving scenes under weather and lighting conditions on different vehicles, the situation of insufficient Corner Case (rare scene) data in actual collection during the research and development test phase can be compensated. The system design takes into account the diversity of data and complex driving scenarios, and uses autonomous algorithms to subscribe to the data in the mass production phase on demand to improve data richness.

[0164] The data collection optimization method and system provided by the disclosed embodiment realize dynamic data priority allocation, can dynamically adjust the data collection strategy according to the model training requirements, and give priority to collecting difficult scene data encountered in model learning, such as driving data under complex intersections and bad weather conditions, to ensure the pertinence and effectiveness of training data. By giving priority to collecting key scene data and optimizing data quality, it promotes the rapid iteration and performance improvement of end-to-end large models, and provides a more solid data foundation for the practical application of intelligent driving technology.

[0165] The data collection optimization method and system provided by the embodiments of the present disclosure construct a distributed data processing architecture, combine cloud computing and edge computing capabilities, realize real-time data processing and analysis, accelerate data flow, reduce data transmission delays, and improve the overall system response speed. By integrating big data processing technologies such as data cleaning, intelligent screening, and automatic labeling, the present invention improves the automation level of data processing, enhances the utilization efficiency of data and the pertinence of model training.

[0166] In addition, it realizes efficient data compression and storage optimization, adopts advanced data compression algorithms to reduce storage space requirements without affecting data integrity and recovery quality. Through intelligent indexing and data tiered storage strategies, it reduces long-term data storage costs and improves data retrieval efficiency. It optimizes efficiency and cost, effectively reduces data storage and processing costs through data compression, efficient storage strategies and distributed processing, while speeding up data processing and improving overall system efficiency.

[0167] Exemplary Devices

[0168] In addition, the present disclosure also provides a data acquisition optimization device, an electronic device, and a computer program product, all of which can be used to implement any data acquisition optimization method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.

[0169] Figure 5 A block diagram of a data acquisition optimization device provided in an embodiment of the present disclosure, the device mainly includes:

[0170] An acquisition module 501 is used to acquire a data collection log, wherein the data collection log includes intelligent driving data of the vehicle collected based on data collection requirements;

[0171] A determination module 502 is used to determine the data subscription amount of each log type according to the data collection log;

[0172] An optimization module 503, used to obtain a data subscription strategy optimized according to the data subscription amount of each log type;

[0173] The adjustment module 504 is used to adjust the data collection requirement based on the optimized data subscription strategy.

[0174] Each module in the above-mentioned model matching device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0175] Exemplary Devices

[0176] Figure 6 A block diagram of an electronic device provided in an embodiment of the present disclosure.

[0177] Reference Figure 6 An embodiment of the present disclosure provides an electronic device, which includes: at least one processor 601; at least one memory 602, and one or more I / O interfaces 603, connected between the processor 601 and the memory 602; wherein the memory 602 stores one or more computer programs that can be executed by the at least one processor 601, and the one or more computer programs are executed by the at least one processor 601 so that the at least one processor 601 can execute the above-mentioned data acquisition optimization method.

[0178] Each module in the above electronic device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0179] Exemplary computer program products and storage media

[0180] The embodiment of the present disclosure also provides a computer program product, including a computer program, which implements the above-mentioned data acquisition optimization method when executed in a processor.

[0181] The computer program may be stored in a readable storage medium of a computer device or in the cloud; the processor of the computer device reads the computer program from the readable storage medium or in the cloud.

[0182] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0183] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0184] As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0185] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0186] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0187] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0188] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0189] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0190] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0191] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0192] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A data acquisition optimization method, characterized in that: include: Acquire a data collection log, wherein the data collection log includes intelligent driving data of the vehicle collected based on data collection requirements; Determining a data subscription amount for each log type according to the data collection log; Obtaining a data subscription strategy optimized according to the data subscription amount of each of the log types; The data collection requirement is adjusted based on the optimized data subscription strategy.

2. The data acquisition optimization method according to claim 1, characterized in that: The data subscription amount of each log type is determined according to the data collection log, including: For each of the data collection logs, determine the similarity between the data collection log and a log template of each log type, and determine the log type to which the data collection log belongs according to the similarity corresponding to each log type; For each of the log types, the number of the data collection logs to which the log type belongs is counted, and the data subscription quantity of the log type is determined according to the counted number.

3. The data acquisition optimization method according to claim 2, characterized in that: Determining the similarity between the data collection log and the log template of each log type includes: For each log type, the log template is processed as follows: Compare the i-th marked data segment in the log template with the i-th marked data segment in the data collection log set, and generate a similarity representation value corresponding to the i-th marked data segment according to the comparison result; i is an integer greater than or equal to 1 and less than or equal to N, and N is the total number of marked data segments included in the log template; The similarity between the data collection log and the log template is determined according to the similarity representation value corresponding to each of the marked data segments in the log template.

4. The data acquisition optimization method according to claim 3, characterized in that: Generating the similarity representation value corresponding to the i-th marked data segment according to the comparison result includes: When the comparison result is equal, determining the similarity representation value corresponding to the i-th marked data segment to be a first preset value; When the comparison result is unequal, the similarity representation value corresponding to the i-th marked data segment is determined to be a second preset value.

5. The data acquisition optimization method according to claim 3, characterized in that: Determining the similarity between the data collection log and the log template according to the similarity representation value corresponding to each of the marked data segments in the log template includes: Determine a sum of similarity representation values ​​corresponding to each of the marked data segments in the log template; The ratio of the obtained sum value to the N is used as the similarity between the data collection log and the log template.

6. The data acquisition optimization method according to claim 2, characterized in that: The determining the log type to which the data collection log belongs according to the similarity corresponding to each log type includes: For each of the log types, when the similarity corresponding to the log template exceeds a similarity threshold corresponding to the log type, the log type is determined as the log type to which the data collection log belongs.

7. A data acquisition optimization system, characterized in that: Including vehicle-side devices and cloud devices; The vehicle-side device is used to collect the intelligent driving data of the vehicle based on data collection requirements and upload it to the cloud device; The cloud device is used to obtain a data collection log, wherein the data collection log includes intelligent driving data of the vehicle collected based on data collection requirements; determine the data subscription amount of each log type according to the data collection log; obtain a data subscription strategy optimized according to the data subscription amount of each log type; and adjust the data collection requirements based on the optimized data subscription strategy.

8. A data acquisition optimization device, characterized in that: include: An acquisition module, used to acquire a data acquisition log, wherein the data acquisition log includes intelligent driving data of the vehicle collected based on data acquisition requirements; A determination module, used to determine the data subscription amount of each log type according to the data collection log; An optimization module, used to obtain a data subscription strategy optimized according to the data subscription amount of each log type; An adjustment module is used to adjust the data collection demand based on the optimized data subscription strategy.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores at least one computer program executable by the at least one processor, and the at least one computer program is executed by the at least one processor so that the at least one processor can perform the data acquisition optimization method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed in a processor, the method for optimizing data collection according to any one of claims 1 to 6 is implemented.

Images