Internet-of-vehicles equipment privacy disclosure analysis method based on large language model

Abstract

The invention discloses an Internet of Vehicles equipment privacy leakage analysis method based on a large language model, which comprises the following steps of: defining a list containing vehicle sensitive data items as an identification reference, establishing a WIFI and Bluetooth communication environment, reconstructing a complete URL (Uniform Resource Locator) and extracting message body data; carrying out preliminary screening on the analyzed network traffic of the Internet of Vehicles equipment by adopting pattern matching, and carrying out deep semantic recognition by utilizing a large language model subjected to parameter fine tuning; visual related privacy information contained in multimedia files contained in the analyzed flow is extracted; according to the method, the privacy policy text of the Internet of Vehicles equipment is analyzed, the entity and behavior of the privacy declaration are extracted, consistency comparison is carried out on the privacy transmission behavior in the identified actual traffic and the privacy policy declaration, and a compliance detection report is generated, the large language model after fine adjustment is utilized, and the identification accuracy of unknown and fuzzy fields is remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of network security and vehicle networking technology, specifically relating to a method for analyzing privacy leaks in vehicle networking devices based on a large language model. Background Technology

[0002] With the widespread adoption of intelligent connected vehicles, devices such as OBD diagnostic tools, in-vehicle cameras, and in-vehicle infotainment systems have become key nodes for data interaction. While providing services, these devices collect a large amount of sensitive information involving user privacy. However, conducting privacy compliance analysis on connected vehicle devices faces significant challenges. First, connected vehicle devices use multiple communication protocols, and the data formats contained in their traffic are diverse, making it difficult for traditional rule-based detection methods to cover all variations. Second, semantic understanding is difficult. Much privacy information related to connected vehicles is hidden in unstructured fields or custom parameters of network messages, and traditional tools lacking contextual understanding capabilities are prone to high false positive or false negative rates. There is also the issue of complex compliance verification. Privacy policies for connected vehicle devices often consist of complex legal texts, and manually comparing the consistency between actual device traffic and policy statements is time-consuming and error-prone, necessitating the introduction of automated privacy compliance analysis methods. Summary of the Invention

[0003] To achieve the above objectives, the technical solution of the present invention is as follows: A method for privacy leakage analysis of vehicle networking devices based on a large language model, the method comprising the following steps: Step one is the classification system construction stage, which defines a list of four categories of sensitive data items, including vehicle data, vehicle environment data, in-vehicle user information, and vehicle network device information, as the identification benchmark; Step two is the mobile data traffic parsing stage, which involves establishing a WIFI and Bluetooth communication environment, intercepting network data packets that interact between the vehicle network device and the cloud or mobile client, performing syntax parsing and semantic restoration of the data packets according to the protocol specifications, reconstructing the complete URL, and extracting the message body data. Step three is the privacy information identification stage. The parsed network traffic of the vehicle networking device is initially screened using pattern matching and deep semantic recognition is performed using a large language model with fine-tuned parameters. For the multimedia files contained in the parsed traffic, the visual-related privacy information contained therein is extracted using object detection model and OCR technology. Step four is the privacy compliance testing phase. The privacy policy text of the vehicle-to-everything (V2X) device is parsed, the privacy statement entities and behaviors are extracted, and the privacy transmission behaviors in the identified actual traffic are compared with the privacy policy statement to generate a compliance testing report.

[0004] Preferably, in the second step of traffic collection and parsing, for HTTPS traffic, a man-in-the-middle proxy is deployed and a root certificate is installed to decrypt the traffic; for Bluetooth traffic, the Bluetooth communication data of the OBD vehicle networking device is obtained using the HCI logs of the mobile client. In traffic parsing, the edit distance between the URLs of HTTP / HTTPS traffic is calculated, and URLs with similarity exceeding a preset threshold are clustered and merged. At the same time, for binary protocols, semantic messages are transformed and extracted using the official protocol documentation.

[0005] Preferably, in the privacy information identification step three, for the text information contained in the vehicle network traffic, a strategy combining pattern matching and a large language model is adopted. First, regular expressions are used to match privacy information with a fixed format. For unstructured or context-dependent fields in the network traffic, they are input into the large language model to identify privacy information. The large language model is pre-tuned using low-rank adaptation technology and the Mind Chain dataset.

[0006] Preferably, the fine-tuning step includes: first, constructing a training dataset with a thought chain annotation format based on a small number of manually labeled traffic samples and corresponding privacy information type samples, where each sample contains input traffic features, intermediate inference steps, and corresponding privacy information type outputs; then, employing low-rank adaptation technology, freezing the original parameters of the pre-trained model, and injecting trainable low-rank matrices only into the model's attention layer or feedforward network layer, updating the weights to the product of two low-dimensional matrices by decomposing them, significantly reducing the number of fine-tuning parameters. During training, the model learns the step-by-step inference path from traffic features to privacy information types based on the thought chain dataset, optimizing the low-rank adapter parameters, and ultimately achieving an efficient, lightweight privacy information recognition model with strong generalization ability.

[0007] Preferably, in the privacy information identification step three, an external knowledge base and prompt word engineering are introduced. Based on the device type, highly relevant privacy information categories are retrieved from the knowledge base, and prompt words containing role settings, task instructions, and small sample examples are constructed to guide the large model to output structured privacy classification results.

[0008] Preferably, for the multimedia information contained in the traffic, the binary data in the message body is deserialized, the panoramic driving perception network is used to detect sensitive targets in the image, the optical character recognition technology is used to extract sensitive text in the image, and it is determined whether the transmission process is encrypted.

[0009] Preferably, in the privacy compliance detection stage of step four, natural language processing technology is used to process the privacy policy text, identify key verbs, extract data type, processing behavior, and receiving entity triples, define three states: compliance, conflict, and non-disclosure, and logically compare the actual detected network data stream tuples with the policy statement tuples to determine whether there is a violation.

[0010] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention utilizes a finely tuned large language model, which can significantly improve the recognition accuracy of unknown and fuzzy fields; through integrated visual analysis, it can discover new privacy risks such as video thumbnail leakage and navigation screenshot leakage; and it achieves full-process automation from traffic capture to compliance determination, effectively reducing the cost of manual auditing. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating the method architecture of an embodiment of the present invention. Detailed Implementation

[0012] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Example

[0013] To address the aforementioned issues, this invention proposes a large language model-assisted method for privacy compliance analysis of vehicle data streams.

[0014] This method comprises four core processing stages: Classification system construction: A list of more than 300 sensitive data items in four categories, including vehicle data, vehicle environment data, in-vehicle user information, and vehicle networking device information, is manually defined and used as the identification benchmark.

[0015] Traffic Collection and Analysis. Passive packet capture technology is used to comprehensively collect Wi-Fi and Bluetooth traffic. HTTP / HTTP2 and OBD protocol data are processed through a protocol parser, and a distance-editing algorithm is used to deduplicate and merge similar URLs, extracting the header, body, and URL parameters. Simultaneously, based on the official OBD protocol documentation, semantic information about the vehicle sensor data contained within is inferred.

[0016] Privacy information identification. Regular expression matching is used for text information in connected vehicle traffic; for complex fields, DeepSeek and other base models are used, combined with LoRA fine-tuning technology and thought chain data for training, enabling the model to possess the reasoning ability of a security expert and accurately identify privacy semantics in the traffic. For image data transmitted in the traffic, the images are first restored, then the YOLOP model is used to identify environmental privacy, and OCR is used to identify text privacy.

[0017] Privacy compliance verification. This function automatically compares identified privacy breaches with privacy policy terms analyzed using NLP technology, generating a compliance report. Example

[0018] This embodiment provides a method for privacy compliance analysis of vehicle data streams assisted by a large language model, including: Module 1: Building a List of Privacy Information for Connected Vehicles

[0019] Based on industry white papers and academic literature, this invention constructs a four-level classification system that includes vehicle data, vehicle environmental data, in-vehicle user information, and device information, covering 327 sensitive data items, as a labeling system for subsequent large-scale model recognition. See Table 1 below.

[0020] Table 1

[0021] Module 2: Traffic Collection and Analysis.

[0022] Passive packet capture technology is used to comprehensively collect Wi-Fi and Bluetooth traffic. HTTP / HTTP2 and OBD protocol data are processed using a protocol parser. The specific steps are as follows: Step 1: Collect device network traffic.

[0023] During traffic collection, different device functions are executed according to a preset script to ensure coverage of major business scenarios. Simultaneously, this invention employs two methods to collect Wi-Fi and Bluetooth traffic generated by the device.

[0024] WiFi Traffic Collection: Set up a WiFi hotspot and enable connected vehicle devices to connect to it. Deploy a man-in-the-middle proxy and import its certificate to intercept and decrypt HTTP / HTTPS traffic between the device and the cloud, recording request headers, response headers, and message bodies.

[0025] Bluetooth Traffic: For devices that can communicate with a smartphone app via Bluetooth, enable the smartphone's HCI logging function to capture Bluetooth underlying communication data packets.

[0026] Step 2: Traffic syntax and semantic analysis.

[0027] First, the data packets captured by Wi-Fi are parsed. For HTTP / HTTPS messages, the URL, Method, Headers, and Body are extracted according to the HTTP standard. Based on this, the URLs in the network stream are reconstructed and merged: the Host, Path, and Query parameters are concatenated. The edit distance of different URLs is calculated; if the similarity is greater than 0.5, they are considered to be of the same functional interface and merged, retaining the one with the most complete parameters, thereby reducing the token overhead required for processing large models. Furthermore, key-value pairs are extracted from the JSON / XML data in the Body for easy identification of privacy information.

[0028] Semantic parsing of messages from vehicle networking devices: Parsing specific vehicle parameters from OBD protocol data based on the PID formula; Module 3: Privacy Information Identification Based on Large Models and Visual Technologies

[0029] 1. Pattern Matching-Based Identification of Vehicle-to-Everything (V2X) Privacy Information. Since V2X devices may directly send vehicle privacy information to remote servers or process the information using various encoding methods before sending, we first search and match predefined privacy information and its encoding results within the traffic to quickly determine if there is any privacy information leakage. Simultaneously, we use predefined regular expressions to quickly identify standard privacy data.

[0030] 2. Vehicle Network Privacy Information Identification Based on Fine-tuned Large Model. In one embodiment of the present invention, the large language model is a pre-trained language model. To make the model suitable for the privacy information identification task in the network traffic of vehicle network devices, the present invention uses a parameter-efficient fine-tuning method to perform targeted training on the model, specifically including the following steps: 1) Freeze the basic parameters of the pre-trained model and introduce trainable low-rank adaptation parameters only in the attention layer and feedforward network layer of the model; 2) Construct a parameter update module based on low-rank adaptation, constrain the weight update through low-rank matrix factorization, and realize the rapid adaptation of the model to the Internet of Vehicles privacy recognition task without changing the original structure of the model. 3) Construct a training sample for privacy identification of vehicle network traffic. The training sample includes at least (a) network traffic input data, which represents the network communication content of the vehicle network device to be analyzed. The input data includes the complete request address and the parameter information it carries; (b) a description of the privacy judgment reasoning process, which records the step-by-step analysis process for the network traffic input data. The reasoning process includes at least: semantic interpretation of the purpose of the network request, decomposition analysis of the names and values ​​of each parameter in the request, and explanation of the judgment basis for aligning the parameter semantics with the predefined privacy information classification standard; (c) privacy information identification results, which represent the privacy information category corresponding to each parameter in the network traffic. The identification results are represented in the form of a mapping relationship between parameter names and privacy categories.

[0031] 4) Based on the training samples, the low-rank adaptation parameters are trained, enabling the model to learn the mapping relationship between field semantics, contextual information, and privacy categories. Through this method, efficient fine-tuning of a large language model is achieved under limited labeled sample conditions, giving it professional reasoning capabilities for privacy information related to vehicle network traffic.

[0032] In the process of privacy information identification, for fields whose privacy type cannot be determined by pattern matching, this invention calls a finely tuned large language model by constructing standardized prompt words. Specifically, it includes: (1) organizing the key name, value and network message context of the field to be analyzed into structured input text; (2) embedding the structured input into a predefined prompt word template to form model input; (3) calling the finely tuned large language model and making inference judgments on whether the field belongs to privacy information based on the prompt word content; (4) parsing the model output results and extracting privacy information category labels as the final identification result.

[0033] The prompt words include at least the following components: (1) a task instruction section, used to clearly inform the large language model that the current analysis target is privacy information identification; (2) a role constraint section, used to limit the model to reason as a privacy analysis expert; (3) a privacy information classification constraint section, used to provide the model with a set of privacy information categories related to the current device type; (4) an input data section, used to carry the network traffic field to be analyzed and its context information; and (5) an output format constraint section, used to specify the format of the model output results so that the system can perform structured parsing. 3. Privacy Information Identification Based on Image Analysis. During network communication in vehicle-to-everything (V2X) devices and their supporting applications, some privacy information is transmitted and displayed in image form. These images may contain sensitive content related to user identity, vehicle identity, or location information. To address the difficulty of identifying privacy information in images solely through network traffic field analysis, this invention proposes a privacy information identification method based on image analysis. Specifically, the method includes the following steps: (1) Filter network response data of image type, select data with image content type from network traffic, and deserialize them to obtain image data to be analyzed; (2) Perform target detection analysis on the image data, use the target detection model to scan the image, locate and mark sensitive target areas in the image that may be associated with privacy information, the sensitive targets include visual objects that can reflect the identity of the user or vehicle, such as human figures and vehicle appearances. (3) Determine the key analysis area based on the target detection results, and obtain the area location corresponding to the sensitive target for subsequent privacy information identification and processing; (4) Perform text content recognition on the image data, detect and recognize the text region in the image, and convert the text information in the image into processable text data; (5) Perform privacy judgment on the identified text information based on predefined rules, and filter out text content with high privacy risks such as address information, location information, identity identifiers or contact information through regular expression matching or pattern recognition. (6) Combine the target detection results and text recognition results to determine privacy information. If the sensitive target is detected in the image or the recognized text information belongs to a predefined privacy information category, then the image is determined to contain privacy information.

[0034] By combining object detection technology with text recognition technology, the collaborative recognition of explicit and implicit privacy information in images in the context of vehicle networking is achieved, thereby improving the completeness and accuracy of image privacy information recognition.

[0035] Module 4: Privacy Compliance Testing.

[0036] This module takes the output of the privacy information identification module and the corresponding privacy policy text of the vehicle-to-everything (V2X) device as input to automatically detect the compliance of privacy data processing behaviors that occur during the actual operation of the V2X device. The privacy policy text is a publicly released privacy policy document from the device manufacturer or its supporting applications. The output of the privacy information identification module includes: the type of privacy data, the corresponding data fields or content, and the destination information for the transmission of this privacy data.

[0037] Since privacy policies are usually described in unstructured natural language and involve various behaviors such as data collection, use, transmission and sharing, it is difficult to directly compare them with specific data transmission behaviors observed in network traffic. This invention introduces natural language processing technology to perform structured parsing of privacy policy text, thereby achieving automated privacy compliance determination.

[0038] Specifically, the privacy compliance detection module includes the following steps: 1) Privacy policy text preprocessing and sentence segmentation, The obtained privacy policy text is preprocessed, including removing formatting marks, normalizing punctuation, and filtering irrelevant content. The privacy policy text is then divided into several semantically complete sentences or clauses to provide basic corpus for subsequent information extraction.

[0039] 2) Extraction of privacy statement elements. For the segmented privacy policy statement, natural language processing technology is used to identify key semantic elements related to privacy data processing, including but not limited to: Privacy data types are used to describe the data content involved in policies; Data processing behavior describes the actions performed on private data; Data receiving object, used to describe the recipient or object of privacy data.

[0040] By extracting the above elements, the natural language descriptions in the privacy policy are transformed into structured privacy statement information units to represent the data processing behaviors explicitly stated in the policy.

[0041] 3) Modeling actual privacy transmission behavior. The privacy data transmission results output by the privacy information identification module are structured and modeled, representing each actual privacy data transmission behavior as a factual information unit containing the following elements: the actual type of privacy data transmitted, the actual data processing behavior, and the actual transmission destination or recipient. These factual information units characterize the actual privacy data processing behavior that occurs during the operation of the vehicle-to-everything (V2X) device.

[0042] 4) Privacy compliance comparison and judgment. The privacy statement information unit extracted in step two is automatically compared with the actual privacy transmission fact information unit constructed in step three. The compliance of privacy data processing behavior is determined based on the comparison results, specifically including: Compliant: When there is an explicit statement in the privacy policy that allows the corresponding privacy data type to be processed within the same or consistent processing behavior and receiving scope, it is deemed compliant; Conflict: A conflict is defined as when a privacy policy explicitly restricts or prohibits a certain type of privacy data processing, and the actual observed behavior exceeds or violates that statement. Undisclosed: When no explicit statement relating to actual privacy data processing is found in the privacy policy text, it is deemed undisclosed.

[0043] 5) Output of compliance testing results. Based on the above judgment results, a structured privacy compliance test report is generated. The report includes at least the privacy data type, the transmission destination, and the corresponding compliance judgment status, which serves as a basis for subsequent privacy risk assessment and compliance audit.

[0044] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A method for analyzing privacy leaks in vehicle-to-everything (V2X) devices based on a large language model, characterized in that, The method includes the following steps: Step one is the classification system construction stage, which defines a list of four categories of sensitive data items, including vehicle data, vehicle environment data, in-vehicle user information, and vehicle network device information, as the identification benchmark; Step two is the mobile data traffic parsing stage, which involves establishing a WIFI and Bluetooth communication environment, intercepting network data packets that interact between the vehicle network device and the cloud or mobile client, performing syntax parsing and semantic restoration of the data packets according to the protocol specifications, reconstructing the complete URL, and extracting the message body data. Step three is the privacy information identification stage. The parsed network traffic of the vehicle networking device is initially screened using pattern matching and deep semantic recognition is performed using a large language model with fine-tuned parameters. For the multimedia files contained in the parsed traffic, the visual-related privacy information contained therein is extracted using object detection model and OCR technology. Step four is the privacy compliance testing phase. The privacy policy text of the vehicle-to-everything (V2X) device is parsed, the privacy statement entities and behaviors are extracted, and the privacy transmission behaviors in the identified actual traffic are compared with the privacy policy statement to generate a compliance testing report.

2. The method according to claim 1, characterized in that, In step two, traffic collection and parsing, for HTTPS traffic, a man-in-the-middle proxy is deployed and a root certificate is installed to decrypt the traffic; for Bluetooth traffic, the Bluetooth communication data of the OBD vehicle networking device is obtained using the HCI logs of the mobile client. In traffic parsing, the edit distance between URLs in HTTP / HTTPS traffic is calculated, and URLs with similarity exceeding a preset threshold are clustered and merged. At the same time, for binary protocols, semantic messages are transformed and extracted using the official protocol documentation.

3. The method according to claim 2, characterized in that, In the third step of privacy information identification, for text information contained in vehicle network traffic, a strategy combining pattern matching and large language model is adopted. First, regular expressions are used to match privacy information with fixed formats. For unstructured or context-dependent fields in network traffic, they are input into the large language model to identify privacy information. The large language model is pre-tuned using low-rank adaptation technology and the MindChain dataset.

4. The method according to claim 3, characterized in that, The fine-tuning steps include: First, based on a small number of manually labeled traffic samples and corresponding privacy information type samples, a training dataset with a mind chain annotation format is constructed. Each sample contains input traffic features, intermediate inference steps, and corresponding privacy information type outputs. Then, a low-rank adaptation technique is used to freeze the original parameters of the pre-trained model and inject trainable low-rank matrices only into the model's attention layer or feedforward network layer. The weights are then updated to the product of two low-dimensional matrices by decomposing them. During training, the model learns the step-by-step inference path from traffic features to privacy information types based on the mind chain dataset, optimizes the low-rank adapter parameters, and finally achieves an efficient, lightweight privacy information recognition model with strong generalization ability.

5. The method according to claim 3, characterized in that, In step three, privacy information identification, an external knowledge base and prompt word engineering are introduced. Based on the device type, highly relevant privacy information categories are retrieved from the knowledge base, and prompt words containing role settings, task instructions, and small sample examples are constructed to guide the large model to output structured privacy classification results.

6. The method according to claim 3, characterized in that, For multimedia information contained in the traffic, the binary data in the message body is deserialized, the panoramic driving perception network is used to detect sensitive targets in the image, the optical character recognition technology is used to extract sensitive text in the image, and it is determined whether the transmission process is encrypted.

7. The method according to claim 1, characterized in that, In the fourth step of the privacy compliance detection stage, natural language processing technology is used to process the privacy policy text, identify key verbs, extract data type, processing behavior, and receiving entity triples, define three states: compliance, conflict, and non-disclosure, and logically compare the actual detected network data stream tuples with the policy statement tuples to determine whether there is a violation.

Images