PCIE-based intelligent driving car desensitization video data bypass collection system and method

Abstract

The application relates to the technical field of intelligent driving cars, and discloses a PCIE-based intelligent driving car desensitization video data bypass collection system, wherein original camera data output by an on-board camera is transmitted to a collection system and an on-board ECU through two independent transmission paths; a lower computer subsystem of the collection system comprises a data collection host, a PCIE video collection card connected with the data collection host and a network attached storage device; a video desensitization algorithm module, a video encoding processing module and a system state module are arranged in the data collection host; an upper computer subsystem of the collection system comprises a data communication module and an event triggering module; through the interaction of the upper and lower computer subsystems, synchronous collection, desensitization, encoding and packaging of the on-board camera data are carried out, NAS storage and an uploading server are simultaneously carried out, real-time updating and sharing of the data are realized, the performance and safety of the intelligent driving car are improved, the privacy of the car owner and other road users is effectively protected, and high-quality data are provided to support autonomous navigation and decision-making of the intelligent driving car.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving vehicle technology, specifically to a PCIE-based intelligent driving vehicle desensitized video data bypass acquisition system and method. Background Technology

[0002] With the development of intelligent driving technology, in-vehicle camera data serves as a training source for intelligent driving algorithms. Major automakers collect vast amounts of in-vehicle camera data during product development. On one hand, this data can be used to improve vehicle design and performance, enhancing the driving experience and safety. On the other hand, it can also be used to research and develop intelligent driving technologies to achieve more autonomous and intelligent vehicles. However, to comply with legal regulations, the collected in-vehicle camera data must undergo anonymization. Data anonymization refers to the transformation of sensitive information, such as license plates and facial information, through anonymization rules to reliably protect sensitive privacy data.

[0003] Existing technologies generally perform data anonymization during in-vehicle data transmission or direct data anonymization during data collection. However, due to limited hardware resources and computing power on the vehicle side, both methods result in problems such as high hardware requirements for data processing, wasted processing capacity, and transmission delays. Summary of the Invention

[0004] The present invention aims to provide a PCIE-based intelligent driving vehicle desensitized video data bypass acquisition system to solve the technical problem of low data desensitization efficiency caused by the high hardware and computing power requirements of existing vehicle data desensitization technologies.

[0005] The basic solution provided by this invention is: a PCIE-based intelligent driving vehicle desensitized video data bypass acquisition system, which operates on intelligent driving vehicles equipped with onboard cameras and onboard ECUs. The system is characterized in that the input terminals of the acquisition system and the onboard ECU are respectively connected to the output terminals of the onboard camera, so that the raw camera data output by the onboard camera is transmitted to the acquisition system and the onboard ECU through two independent transmission paths. The acquisition system includes a lower-level subsystem and a higher-level subsystem, transmitting a data stream with timestamps.

[0006] The lower-level subsystem includes a data acquisition host and a PCIe video capture card and a network auxiliary storage device connected to the data acquisition host. The data acquisition host is equipped with a video desensitization algorithm module, a video encoding processing module, and a system status module. The PCIe video capture card is used to receive and process raw camera data to form YUV data. The video desensitization algorithm module is used to receive YUV data and perform desensitization processing to form desensitized data. The video encoding processing module is used to receive the desensitized data and simultaneously perform first encoding encapsulation and second encoding, wherein the first encoding encapsulation forms first data and sends it to the network auxiliary storage device, and the second encoding forms second data and sends it to the upper-level subsystem. The system status module is used to collect system status data and send it to the upper-level subsystem and the network auxiliary storage device.

[0007] The host computer subsystem includes a data communication module and an event triggering module; the data communication module is used to receive and process second data and system status data sent by the lower computer subsystem; the event triggering module is used to mark events according to the second data and system status data, form marking information, and transmit the marking information to the network auxiliary storage device.

[0008] The network-attached storage device is used to receive and store first data with timestamp association, system status data, and annotation information, and upload them to the server in real time.

[0009] The working principle and advantages of this invention are as follows: Through the interaction between the upper-level computer subsystem and the lower-level computer subsystem, the data of the vehicle camera is synchronously collected, desensitized, encoded and packaged, and simultaneously stored in NAS (Network Attached Storage) and uploaded to the server, realizing real-time data updates and sharing, thereby improving the performance and safety of intelligent driving vehicles; it can effectively protect the privacy of vehicle owners and other road users, while providing high-quality data support for the autonomous navigation and decision-making of intelligent driving vehicles.

[0010] Compared with the prior art, the advantages of the present invention are:

[0011] 1) The traditional method of directly anonymizing data between the vehicle camera and the vehicle ECU can, on the one hand, interfere with the communication between the camera and the ECU, thereby affecting the operation of the vehicle decision-making algorithm and causing potential safety hazards; on the other hand, it directly uses the vehicle system resources for a large amount of data processing, which increases the load and operating resource consumption of the vehicle system.

[0012] This invention uses a two-way bypass to collect camera data from the camera output end. Without interfering with the normal operation of the vehicle, the vehicle body data can be desensitized, encoded, encapsulated, and stored before being directly uploaded to the cloud for algorithm development teams to optimize algorithms and upgrade the system. The branch layout reduces the computing power and hardware requirements of the vehicle itself, does not occupy ECU resources, and improves the robustness and stability of the vehicle system.

[0013] 2) Traditional methods for desensitizing video data, the processing flow is as follows: Figure 1 As shown, in traditional methods, the camera is directly connected to the ECU. If data anonymization and compression are performed during transmission, it increases the time for video data to be transmitted to the ECU, affecting the normal operation of the vehicle. Therefore, video data must be encoded and stored first, and anonymization is performed after subsequent acquisition. When storing YUV data, encoding is required. H.264 / H.265 encoding compresses the UV components to a certain extent to reduce the bitrate. Since the UV components reflect color saturation and hue, the compression process causes color distortion or blurring in the video. In existing methods, before anonymizing the video data after it is stored on the disc, the video file must be decoded into YUV image data. At this point, the UV components in the YUV image data have already undergone compression processing during the disc storage. During the process of encoding the anonymized data into H.264 / H.265 and encapsulating it into an MP4 video file, the UV components are compressed again, causing secondary damage to the image. This method involves two encoding operations on the YUV data, affecting the accuracy and robustness of the anonymization algorithm, resulting in a serious deterioration in the quality of the stored image.

[0014] This solution employs a one-to-two bypass camera acquisition system. Camera data can be simultaneously transmitted to the ECU and the data acquisition host. The data acquisition host can asynchronously perform de-identification and encoding processing on the camera data without interfering with the vehicle's normal operation. YUV data acquired from the camera is directly de-identified, then encoded and stored. Figure 2 As shown, the entire process involves only one encoding operation. On the one hand, this simplifies the data desensitization and storage process; on the other hand, it avoids color distortion or blurring of UV components during compression, improving the accuracy and robustness of the desensitization algorithm and enhancing video quality.

[0015] 3) Currently, most manufacturers use offline algorithms for scene extraction. Firstly, this requires high-performance servers, which is costly. Secondly, the accuracy of the algorithm itself cannot be guaranteed, and the recognition of different weather and environmental conditions needs improvement. Traditional data collection methods do not label information such as camera operation status, vehicle traffic conditions, and weather conditions during the collection process, which is not conducive to data analysis and filtering by algorithm development teams.

[0016] This solution uses a two-branch process to process the de-identified data: one branch stores the video files, while the other provides real-time preview. It also annotates various scenarios during vehicle operation, enabling the association between the collected data and the scenarios. This allows the R&D team to select the corresponding scenario data for algorithm training.

[0017] This invention provides a PCIE-based intelligent driving vehicle desensitized video data bypass acquisition system, and also a PCIE-based intelligent driving vehicle desensitized video data bypass acquisition method to solve the technical problem of low data desensitization efficiency caused by the high hardware and computing power requirements of existing vehicle data desensitization technologies.

[0018] The method includes the following steps:

[0019] S1: The raw camera data output by the vehicle camera is transmitted to the PCIE video capture card and the vehicle ECU through two independent transmission paths respectively.

[0020] S2: The PCIe video capture card receives raw camera data and deserializes it to form YUV data, which is then transmitted to the data acquisition host. The data acquisition host provides time synchronization to the PCIe video capture card through a time synchronization device.

[0021] S3: The data acquisition host receives YUV data and performs desensitization to form desensitized data; the data acquisition host performs first encoding and encapsulation on the desensitized data to form first data;

[0022] S4: The data acquisition host synchronously performs a second encoding on the de-identified data to form second data; the data acquisition host collects system status data and synchronously sends the second data and system status data to the host computer subsystem;

[0023] S5: The host computer subsystem receives the second data and system status data, performs real-time preview display and event trigger annotation, and sends the annotation information generated by the event trigger annotation to the data acquisition host;

[0024] S6: The first data, system status data, and annotation information are synchronously stored in the network auxiliary storage device through the data acquisition host, and then uploaded to the server in real time through the network auxiliary storage device.

[0025] Beneficial effects: It breaks through the conventional approach of combining in-vehicle data testing and in-vehicle applications, reducing the demand on in-vehicle computing power and hardware through independent branch transmission, while ensuring that the two data streams do not interfere with each other. This significantly reduces the need for the vehicle to identify the specific content of the data, improves transmission speed, and enhances the robustness and stability of the vehicle system. It simplifies the data anonymization and storage process, avoids data compression, and improves data quality. It establishes a correlation between data and scenarios, which is beneficial for algorithm development teams to analyze and filter data, thereby increasing the value of data utilization. It achieves time synchronization between the system and the data collection, which is beneficial for the use of collected data in training related algorithms such as target fusion. Attached Figure Description

[0026] Figure 1 A flowchart for traditional data anonymization;

[0027] Figure 2 The flowchart of the desensitization process of the PCIE-based intelligent driving vehicle desensitized video data bypass acquisition system provided in the embodiment of the present invention is shown.

[0028] Figure 3 This is a schematic diagram of the PCIE-based intelligent driving vehicle desensitized video data bypass acquisition system provided in an embodiment of the present invention.

[0029] Figure 4 The flowchart illustrates the PCIE-based intelligent driving vehicle desensitized video data bypass acquisition method provided in this embodiment of the invention.

[0030] Figure 5 This is a diagram of the camera configuration form of the PCIE-based intelligent driving vehicle desensitized video data bypass acquisition system provided in an embodiment of the present invention.

[0031] Figure 6 This is a network model diagram of the PCIE-based intelligent driving vehicle desensitized video data bypass acquisition system provided in an embodiment of the present invention.

[0032] Figure 7 This is a flowchart illustrating the desensitization model training and deployment process of the PCIE-based intelligent driving vehicle desensitized video data bypass acquisition system provided in this embodiment of the invention.

[0033] Figure 8 The flowchart of the video desensitization process of the PCIE-based intelligent driving vehicle desensitized video data bypass acquisition system provided in the embodiment of the present invention;

[0034] Figure 9 This is a scene marking panel diagram of the PCIE-based intelligent driving vehicle desensitized video data bypass acquisition system provided in an embodiment of the present invention. Detailed Implementation

[0035] The following detailed explanation illustrates the specific implementation methods:

[0036] The basic implementation examples are as follows: Figure 3 As shown: A PCIE-based intelligent driving vehicle desensitized video data bypass acquisition system is used in intelligent driving vehicles equipped with onboard cameras and onboard ECUs. The input terminals of the acquisition system and the onboard ECU are respectively connected to the output terminals of the onboard cameras, so that the raw camera data output by the onboard cameras is transmitted to the acquisition system and the onboard ECU through two independent transmission paths. The acquisition system includes a lower-level subsystem and a higher-level subsystem, which transmits data streams with timestamps.

[0037] The lower-level subsystem includes a data acquisition host and a PCIE video capture card and a network auxiliary storage device connected to the data acquisition host. Six 800W pixel GSML cameras installed at different locations on the vehicle body (front view, rear view, side view, etc.) are connected to the integrated PCIE video capture card. The data acquisition host includes a video desensitization algorithm module, a video encoding processing module, and a system status module. The PCIE video capture card receives and processes raw camera data to form YUV data. The video desensitization algorithm module receives YUV data and performs desensitization processing to form desensitized data. The video encoding processing module receives the desensitized data and simultaneously performs first encoding encapsulation and second encoding. The first encoding encapsulation forms first data, which is sent to the network auxiliary storage device, and the second encoding forms second data, which is sent to the upper-level subsystem. The system status module collects system status data and sends it to the upper-level subsystem and the network auxiliary storage device.

[0038] The host computer subsystem includes a data communication module and an event triggering module; the data communication module is used to receive and process the second data and system status data sent by the lower computer subsystem; the event triggering module is used to annotate events according to the second data and system status data, form annotation information, and transmit the annotation information to the network auxiliary storage device.

[0039] The network-attached storage device is used to receive and store first data with timestamp association, system status data, and annotation information, and upload them to the server in real time.

[0040] The host computer subsystem also includes a configuration module and a first instruction processing module. Figure 3 The configuration module is used for setting parameters for instruction interaction between the host computer subsystem and the slave computer subsystem; the first instruction processing module is used for issuing instructions to control the actions of the slave computer subsystem.

[0041] The host computer subsystem also includes an online status monitoring module; the online status monitoring module is used to receive and display the second data and system status data transmitted by the data communication module in real time.

[0042] The data acquisition host also has a second instruction processing module ( Figure 3 The first instruction processing module is used to receive instructions from the first instruction processing module and call the devices connected to the corresponding interfaces of the data acquisition host according to the instructions.

[0043] The lower-level subsystem also includes a PTP switch; the PTP switch is connected between the data acquisition host and the PCIE video capture card, and is used by the data acquisition host to provide time synchronization for the PCIE video capture card.

[0044] The lower-level subsystem also includes a power supply voltage regulator module, which connects to the data acquisition host via a corresponding interface. The lower-level subsystem needs to access multiple high-resolution cameras for data acquisition and processing, and also requires a graphics card for data anonymization. Therefore, it requires a high-performance and stable system to support it. Considering the system's high performance and high power consumption, this solution provides a power supply voltage regulator module to ensure long-term stable operation and effectively avoid various problems caused by unstable voltage in the vehicle's electrical system.

[0045] The main function of the host computer subsystem is to configure the acquisition system and access devices, issue commands to control the actions of the slave computer subsystem, receive and process the data returned by the slave computer subsystem, realize online real-time data preview and system status monitoring, perform event trigger annotation, and issue annotation commands to the slave computer for data annotation.

[0046] The lower-level acquisition program runs on the data acquisition host and is mainly used for the synchronous acquisition, desensitization, encoding and encapsulation of raw camera data. At the same time, it stores the data on the network Attached Storage (NAS) device and uploads it to the server to achieve data sharing.

[0047] Based on the aforementioned PCIE-based intelligent driving vehicle desensitized video data bypass acquisition system, this invention provides a PCIE-based intelligent driving vehicle desensitized video data bypass acquisition method, such as... Figure 4 As shown, this also enables the implementation of a PCIE-based intelligent driving vehicle desensitized video data bypass acquisition system.

[0048] First, prepare for interaction between the upper and lower computer subsystems:

[0049] according to Figure 3As shown, connect the entire data acquisition system equipment, start the host computer and the slave computer. After the host computer and the slave computer are initialized, the host computer starts the Grpc client and the slave computer starts the Grpc server, waiting for the command interaction between the host computer and the slave computer.

[0050] S0: The host computer subsystem configures relevant parameters for acquisition, video processing, and storage via the configuration module. These parameters include image quality, storage resolution, display resolution, frame rate, bitrate, bitrate control mode, encoding method, and container format. Image quality can be set from low to high as Average, Better, and Best. The bitrate control mode can be set to constant bitrate / variable bitrate, and the encoding method can be set to H.264 / H.265. Dynamic settings for encoding parameters are also possible. Figure 5 As shown, this allows the system to adapt to vehicle-mounted cameras of different resolutions, improving the system's versatility.

[0051] The upper and lower computer subsystems exchange instructions to process data.

[0052] The host computer subsystem sends configurations to the slave computer subsystem via the first instruction module (GRPC instructions). It controls the slave computer subsystem's connection and disconnection from the vehicle-mounted camera and video file storage via a set of instructions such as connect, disconnect, start acquisition, and stop acquisition. The slave computer subsystem, through the second instruction processing module, calls different interfaces and the corresponding connected devices to process various instructions sent by the host computer subsystem via the first instruction module. Upon startup, the slave computer subsystem registers GRPC server-side instructions. When the host computer subsystem sends a connection instruction, it creates a GRPC client to connect to the slave computer subsystem's GRPC server, establishing instruction communication between the two subsystems. Simultaneously, it obtains the server address for data transmission from the slave computer subsystem via a sent instruction and creates a data transmission and reception socket connection through the host computer subsystem, enabling data interaction between the two systems. The host computer subsystem ensures data reliability and system stability through real-time data preview and status monitoring. After the device connection is normal, data acquisition begins. After data acquisition is complete, a stop acquisition instruction is sent. If multiple data acquisition segments are required, the above start and stop acquisition process is repeated. After data acquisition is complete, disconnect the lower-level subsystem data acquisition device and restore the initial state.

[0053] The PCIe-based bypass acquisition method for de-identified video data of intelligent driving vehicles also includes:

[0054] S1: The raw camera data output by the vehicle-mounted camera is transmitted to the PCIE video capture card and the vehicle-mounted ECU through two independent transmission paths.

[0055] Specifically, in this process, the raw camera data is deserialized using a deserializer, processed by an FPGA (Field-Programmable Gate Array) module, and then distributed to the PCIe video capture card and the vehicle ECU. The raw camera data is then bypassed from the vehicle camera output via a splitter, allowing for data anonymization and other processing before being uploaded to the cloud for algorithm optimization and system upgrades by the algorithm development team, without consuming ECU resources. This improves the robustness and stability of the vehicle's infotainment system.

[0056] S2: The PCIe video capture card receives raw camera data and deserializes it to form YUV data, which is then transmitted to the data acquisition host. The data acquisition host provides time synchronization to the PCIe video capture card through a time synchronization device.

[0057] Specifically, the video capture card can efficiently output high-quality YUV format video frame data in real time. When the second instruction module receives the connection instruction issued by the first instruction module, it controls the PCIE video capture card to successfully connect to the camera. The YUV data of the multiple vehicle-mounted cameras (vehicle-mounted cameras) output by the video capture card is transmitted to the data acquisition host through the PCIE bus.

[0058] The data acquisition host provides time synchronization to the PCIe video capture card via a PTP switch, ensuring time synchronization between the entire acquisition system and the PCIe video capture card. Simultaneously, the vehicle-mounted cameras employ a TTL synchronization trigger mechanism, with all cameras on the vehicle triggering at the same time, sending timestamps and data to the data acquisition host simultaneously, guaranteeing scene synchronization across all cameras. During data anonymization, the timestamp continues to accompany data storage, ensuring data time synchronization and scene consistency. This synchronization enables the acquired data to be used for training algorithms such as target fusion, enhancing data utilization value.

[0059] S3: The data acquisition host receives YUV data and performs desensitization to form desensitized data;

[0060] Specifically, the data acquisition host obtains YUV image frame data from the PCIe video capture card through the V4L2 interface. Since the acquired data is real road video data, which contains a large amount of private information, such as license plates and faces, it is necessary to anonymize the acquired data to protect its privacy and to meet relevant legal monitoring requirements.

[0061] This solution provides a de-identification algorithm with good real-time performance, high target detection rate, and low false positive and false negative rates, thus improving the quality of de-identification. The de-identification algorithm in this system is based on deep learning, and a state-of-the-art (SOTA) target detection algorithm is selected and modified for license plate and face detection. A large amount of road view video was collected, including data from multiple scenes, various road conditions and weather conditions, and different license plate types for data annotation. Tools such as ffmpeg and OpenCV were used to process the video into multi-frame images of sizes and formats supported by the algorithm, and the Labelme annotation tool was used to annotate the license plates and faces in the images. After obtaining the annotated data, a script was written to convert the format of the annotated data to a format supported by the algorithm model input. Then, training was performed on the PyTorch deep learning framework using a large amount of annotated data. The output layer was modified to only detect valid targets such as faces and license plates. Through data augmentation, modification of the algorithm network model structure, and hyperparameter settings, multiple training and hyperparameter tuning were performed, finally obtaining a training network model with a map value higher than 0.94. Figure 6 As shown.

[0062] To enable the algorithm to function in the data acquisition host environment, the trained algorithm model needs to be deployed in C++. Since the data acquisition host integrates an RTX 3060 graphics card, TensorRT, with its superior inference performance, is chosen for deployment. Figure 7 As shown, firstly, the trained model weights (pt files) are exported as ONNX files using a Python script for deployment. Secondly, based on the current graphics card model of the data acquisition host, the corresponding versions of graphics card drivers, CUDA, cuDNN, TensorRT, etc., are downloaded and installed. TensorRT is then used to convert the ONNX model into a TRT model. Next, a C++ deployment program is written to perform TensorRT deployment inference. For the obtained valid detection target regions, mean blurring is performed using OpenCV, finally forming a complete desensitization processing algorithm module. Testing shows that this algorithm can achieve real-time processing of video data with a resolution of 3840*2160 and a frame rate of 30.

[0063] like Figure 8As shown, the specific data anonymization process is as follows: by default, all data needs to be anonymized. For special scenarios where anonymization is not required, the instruction module directly skips this step and does not receive anonymization instructions. The data acquisition host obtains YUV format video data through the V4L2 interface. First, the video anonymization algorithm module converts the YUV data obtained by the data acquisition host frame by frame into Mat video frame data required by the anonymization algorithm. Second, the entire Mat video frame data is copied to the GPU (Graphics Processing Unit) and then preprocessed, such as adjusting the resolution and size, and performing post-inference processing, such as transforming vector coordinates, to obtain the detection results. By setting thresholds, such as setting a confidence level greater than 0.5 and an IMU threshold greater than 0.45, valid detection targets are obtained. OpenCV is used to anonymize the region where the target is located in the image. Finally, the anonymized Mat video frame data is restored to YUV video frame format for subsequent video encoding and MP4 encapsulation.

[0064] In S4, the data acquisition host synchronously performs first encoding and second encoding on the de-identified data. The first encoding and encapsulation forms first data and sends it to the network auxiliary storage device, while the second encoding forms second data and sends it to the host computer subsystem. The data acquisition host collects system status data and synchronously sends the second data and system status data to the host computer subsystem.

[0065] Specifically, this solution performs a two-branch process on the de-identified data. One branch performs the first encoding and stores the data as a video file, while the other branch performs the second encoding for real-time preview. Event annotations are added to various scenarios during vehicle operation to associate the collected data with the scenarios, making it easier for the R&D team to select the corresponding scenario data for algorithm training.

[0066] The first encoding is H264 / H265 encoding. After receiving the start recording command from the host subsystem, the lower-level subsystem encodes the de-identified video frames using the NVIDIA Video Codec SDK provided by the data acquisition host. First, it obtains the encoding parameters from the host subsystem configuration module, initializes the encoding interface of the NVIDIA Video Codec SDK on the data acquisition host, and finally encodes the de-identified image data into an H264 / H265 bitstream. After receiving the start storage command from the first command module, it calls the relevant interfaces of MP4V2 to encapsulate the H264 / H265 encoded video data into MP4 format to form the first data, which is then stored as a video file for subsequent steps. Currently, it mainly supports H264 / H265 bitstream video data.

[0067] The second encoding is JPEG encoding. Since YUV data is relatively large and not conducive to network transmission, the lower-level subsystem sends YUV data to the upper-level subsystem as image data after JPEG encoding. This is different from the encoding format of the encapsulation format H264 / H265, which is more conducive to transmission.

[0068] System status data includes the on / off status information of the vehicle-mounted camera, network status information of the lower-level subsystem, device online information, remaining disk capacity, and CPU utilization. The upper-level subsystem, through its data communication module, parses the received video data and status information from the lower-level machine, enabling online video monitoring and online system status monitoring of the entire acquisition system and process.

[0069] S5: The host computer subsystem receives the second data and system status data, performs real-time preview display and event trigger annotation, and sends the annotation information formed by the event trigger annotation to the data acquisition host.

[0070] Specifically, the real-time preview display allows for visualization of video and system status through the online video monitoring module, facilitating real-time viewing and event trigger annotation by testers.

[0071] The method for triggering event annotation is as follows: the second data and system status data displayed in the real-time preview are analyzed and judged based on at least the operating status of the vehicle camera, the road conditions and / or the weather conditions. The second data and system status data that meet the annotation conditions are annotated to form annotation information and annotation instructions, which are then sent to the data acquisition host. The data acquisition host associates the annotation information with the corresponding first data and system status data using timestamps according to the annotation instructions.

[0072] The event annotation information is independent and is associated with the data in the data acquisition host through timestamps. That is, the timestamp of the annotation information is used to establish a connection with the first data video file that is finally stored, so that the first data that is annotated is also accompanied by an annotated JSON file.

[0073] Specifically, such as Figure 9 As shown, testers view real-time preview data, including video and text information about system operation, through the host computer subsystem. Figure 9The event trigger panel shown allows manual marking of vehicle operation scenarios, such as heavy rain, heavy snow, sandstorms, dark tunnels, highways, urban roads, and rural roads. This provides more diverse information to the raw camera data, facilitating subsequent algorithm optimization by allowing the team to filter training data based on these scenarios and providing broader reference value during optimization training. Simultaneously, scenarios where camera data quality does not meet requirements can also be marked, such as when the camera is fogged by rain, has low visibility, or is obstructed by foreign objects. For situations with no vehicles or no drivers, commands can be issued through the first command module to perform data collection without anonymization, reducing system resource consumption and allowing for pre-configuration.

[0074] The annotation information can be in the form of: {"timestamp":"1709372938384265","channel":"1","roadlevel":"Highway","weather":"Light rain","temperature":"20",} where timestamp is the timestamp of the image at the moment of triggering, accurate to microseconds; channel is the corresponding camera channel; and temperature is in degrees Celsius.

[0075] It should be noted that the annotation information is a description of the relevant scenario based on timestamps. For an event, its JSON file and the corresponding video file are associated through timestamps. When an event is triggered to generate annotation information, an annotation instruction is sent to the data acquisition host. After receiving the annotation instruction, the data acquisition host associates the annotation information with the corresponding first data and system status data based on the timestamps, and stores the annotation information in JSON format on the NAS device, and stores the corresponding first data as a video file on the NAS device. Data with timestamp associations are stored in parallel.

[0076] S6: The first data, system status data, and annotation information are synchronously stored in the network auxiliary storage device through the data acquisition host, and then uploaded to the server in real time through the network auxiliary storage device.

[0077] Specifically, the first data, system status data, and annotation information are timestamped and linked. The data is written to the configured NAS and uploaded to the server in real time for sharing. The data uploaded to the server or stored in the NAS can be used for training, autonomous navigation, and decision-making of intelligent driving vehicle algorithms. At the same time, because the data is labeled with scenes, the collected data is associated with the scenes, making it convenient for the R&D team to select the corresponding scene data for algorithm training.

[0078] When the lower-level subsystem receives a stop recording command from the upper-level subsystem, it will stop storing video, but will still push data to the upper-level subsystem online. When the lower-level subsystem receives a disconnect command from the upper-level subsystem, it will stop capturing data from the camera and stop pushing data to the upper-level subsystem.

[0079] This embodiment provides a PCIE-based intelligent driving vehicle de-identified video data bypass acquisition system and method, which breaks through the conventional approach of integrating vehicle data testing and acquisition with vehicle applications. By using independent branch transmission, it reduces the demands on vehicle computing power and hardware, while ensuring that the two data streams do not interfere with each other. This significantly reduces the need for vehicle-side data content identification, improves transmission speed, and enhances the robustness and stability of the vehicle system. It also simplifies the data de-identification and storage process, avoids data compression, and improves data quality. Furthermore, it establishes a correlation between data and scenarios, facilitating data analysis and filtering by algorithm development teams and increasing the value of data utilization. Finally, it achieves time synchronization between the system and acquisition, enabling the acquired data to be used for training related algorithms such as target fusion.

[0080] The above descriptions are merely embodiments of the present invention. Commonly known structures and characteristics of the solutions are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent.

Claims

1. A PCIE-based intelligent driving vehicle desensitized video data bypass acquisition system, characterized in that, This system operates on intelligent driving vehicles equipped with onboard cameras and onboard ECUs. The input terminals of the acquisition system and the onboard ECU are respectively connected to the output terminals of the onboard camera, so that the raw camera data output by the onboard camera is transmitted to the acquisition system and the onboard ECU through two independent transmission paths. The acquisition system includes a lower-level subsystem and a higher-level subsystem, which transmits data streams with timestamps. The lower-level subsystem includes a data acquisition host and a PCIE video capture card and network auxiliary storage device connected to the data acquisition host; the data acquisition host is equipped with a video desensitization algorithm module, a video encoding processing module and a system status module; the PCIE video capture card is used to receive and process raw camera data to form YUV data; the video desensitization algorithm module is used to receive YUV data and perform desensitization processing to form desensitized data; The video encoding processing module is used to receive de-identified data and simultaneously perform first encoding encapsulation and second encoding. The first encoding encapsulation forms first data and sends it to the network auxiliary storage device, and the second encoding forms second data and sends it to the host computer subsystem. The system status module is used to collect system status data and send it to the host computer subsystem and the network auxiliary storage device. The host computer subsystem includes a data communication module and an event triggering module; the data communication module is used to receive and process second data and system status data sent by the lower computer subsystem; the event triggering module is used to mark events according to the second data and system status data, form marking information, and transmit the marking information to the network auxiliary storage device. The network-attached storage device is used to receive and store first data with timestamp association, system status data, and annotation information, and upload them to the server in real time.

2. The PCIE-based intelligent driving vehicle desensitized video data bypass acquisition system according to claim 1, characterized in that, The host computer subsystem further includes a configuration module and a first instruction processing module; the configuration module is used for setting parameters for instruction interaction between the host computer subsystem and the slave computer subsystem; the first instruction processing module is used for issuing instructions to control the actions of the slave computer subsystem. The data acquisition host also includes a second instruction processing module; the second instruction processing module is used to receive instructions issued by the first instruction processing module, and to call and operate the devices connected to the corresponding interfaces of the data acquisition host according to the instructions.

3. The PCIE-based intelligent driving vehicle desensitized video data bypass acquisition system according to claim 1, characterized in that, The host computer subsystem also includes an online status monitoring module; the online status monitoring module is used to receive and display the second data and system status data transmitted by the data communication module in real time.

4. The PCIE-based intelligent driving vehicle desensitized video data bypass acquisition system according to claim 1, characterized in that, The system status data includes the on / off status information of the vehicle-mounted camera, the network status information of the lower-level subsystem, the online status information of the device, the remaining disk capacity, and the CPU utilization rate.

5. The PCIE-based intelligent driving vehicle desensitized video data bypass acquisition system according to claim 1, characterized in that, The events mentioned include the operation status of the vehicle-mounted camera, road conditions, and weather conditions.

6. The PCIE-based intelligent driving vehicle desensitized video data bypass acquisition system according to claim 1, characterized in that, The lower-level subsystem also includes a PTP switch; the PTP switch is connected between the data acquisition host and the PCIE video capture card, and is used for the data acquisition host to provide time synchronization for the PCIE video capture card.

7. The PCIE-based intelligent driving vehicle desensitized video data bypass acquisition system according to claim 1, characterized in that, The lower-level subsystem also includes a power supply voltage regulator module connected to the data acquisition host.

8. A PCIE-based method for bypassing the acquisition of desensitized video data for intelligent driving vehicles, characterized in that... The method, applicable to intelligent driving vehicles equipped with onboard cameras and onboard ECUs, includes the following steps: S1: The raw camera data output by the vehicle camera is transmitted to the PCIE video capture card and the vehicle ECU through two independent transmission paths respectively. S2: The PCIe video capture card receives raw camera data and deserializes it to form YUV data, which is then transmitted to the data acquisition host. The data acquisition host provides time synchronization to the PCIe video capture card through a time synchronization device. S3: The data acquisition host receives YUV data and performs desensitization to form desensitized data; S4: The data acquisition host synchronously performs first encoding and second encoding on the de-identified data. The first encoding and encapsulation forms first data and sends it to the network auxiliary storage device. The second encoding forms second data and sends it to the host computer subsystem. The data acquisition host collects system status data and synchronously sends the second data and system status data to the host computer subsystem. S5: The host computer subsystem receives the second data and system status data, performs real-time preview display and event trigger annotation, and sends the annotation information generated by the event trigger annotation to the data acquisition host; S6: The first data, system status data and annotation information are synchronously stored in the network auxiliary storage device through the data acquisition host, and then uploaded to the server in real time through the network auxiliary storage device.

9. The PCIE-based method for bypassing the acquisition of desensitized video data for intelligent driving vehicles according to claim 8, characterized in that, In S4, the first encoding is H264 / H265 encoding, and the second encoding is JPEG encoding.

10. The PCIE-based method for bypassing the acquisition of desensitized video data for intelligent driving vehicles according to claim 8, characterized in that, The event-triggered annotation method in S5 is as follows: the second data and system status data displayed in the real-time preview are analyzed and judged based on at least the vehicle camera operation status, road conditions and / or weather conditions. The second data and system status data that meet the annotation conditions are annotated to form annotation information and annotation instructions, which are then sent to the data acquisition host. The data acquisition host associates the annotation information with the corresponding first data and system status data using timestamps according to the annotation instructions.

Images