Methods for semantic encoding of intelligent driving scenarios on-chain and integration of multiple downstream tasks

Abstract

This invention provides a method for semantic encoding uplink and multi-downstream task integration in intelligent driving scenarios, including: constructing a keyframe feature encoder M for video data in intelligent driving scenarios. e Encoder parameters are learned through a self-supervised reconstruction task; decoders for various downstream tasks, such as security officer abnormal behavior recognition, scene reconstruction, and security officer face recognition, are designed according to the business needs of blockchain shared nodes. i Multiple decoders D are jointly supervised by multiple downstream tasks. i Training; building a framework for on-chain semantic encoding of video data and downstream task applications, covering input video data S, and filtering keyframes S. f Obtain keyframe semantic encoding information M e (S f On-chain, multi-node consensus and encoding information sharing, obtaining downstream task decoding information D i [M e (S f This invention achieves efficient representation and storage of massive and complex data through semantic encoding on the blockchain, meeting the diverse downstream task requirements of blockchain sharing nodes, thereby promoting efficient and reliable storage and sharing of data in related fields.

Description

Technical Field

[0001] This invention relates to the field of blockchain applications, specifically to a method for on-chain semantic encoding of intelligent driving scenarios and integration of multiple downstream tasks. Background Technology

[0002] The collection, processing, and analysis of massive amounts of real-time data are the core driving force for promoting technological optimization and iteration and the widespread application of products in the intelligent connected vehicle industry. However, the effective management and full utilization of large amounts of multi-source sensor data remains a major challenge restricting the further development of the intelligent connected vehicle industry. Furthermore, due to inconsistent data formats, the lack of established trust mechanisms, and the absence of standardized sharing mechanisms, data silos exist between different companies or institutions in the autonomous driving industry, making it difficult to fully explore and utilize the value of massive amounts of data. Traditional centralized intelligent driving data management solutions have high storage capacity requirements and are subject to the risks of data loss, tampering, and leakage. The lack of unified data standards and interoperability leads to low data utilization efficiency and difficulty in data sharing between different companies. Traditional blockchain-based intelligent driving video data storage and sharing solutions are limited by the storage characteristics of blockchain and cannot support the on-chain storage and sharing of massive amounts of real-time data. This invention proposes a method for semantic encoding and blockchain integration of intelligent driving scenarios. By semantically encoding video data in intelligent driving scenarios and uploading it to the blockchain, it achieves efficient representation and storage of massive and complex data while maintaining data integrity and accuracy. This ensures the credibility of the data source while enabling secure data sharing, and meets the diverse downstream task requirements of blockchain sharing nodes, thereby promoting efficient and reliable storage and sharing of data in related fields. Summary of the Invention

[0003] This invention provides a method for semantic encoding and on-chaining of intelligent driving scenarios and integration of multiple downstream tasks, characterized in that the method includes:

[0004] Step 1: Collect intelligent driving scenario video dataset S raw and the original dataset S raw Preprocessing and manual annotation are performed to form a comprehensive, fine-grained manually annotated dataset S for various downstream task scenarios;

[0005] Step 2: Process the input video data S based on the keyframe filtering algorithm to obtain the keyframe set S. f ;

[0006] Step 3: Based on the semantic encoding information of keyframes in video data available for multiple downstream task scenarios, construct a deep learning-based keyframe feature encoder M for intelligent driving scenarios. e ;

[0007] Step 4: Perform keyframe feature encoder M based on the self-supervised intelligent driving scene image reconstruction task. eTraining and parameter learning;

[0008] Step 5, use M e Acquiring key frames S of video data in intelligent driving scenarios f Semantic encoded information M e (S f ), and upload it to blockchain node N. i The encoded information is stored and shared through multi-node consensus on the blockchain.

[0009] Step 6, Blockchain shared node N j Using the keyframe semantic encoding information M stored on the chain e (S f Design downstream task encoder D according to specific business scenarios. i Including but not limited to the safety officer abnormal behavior recognition decoder D1, the scene reconstruction decoder D2, and the safety officer face recognition decoder D3;

[0010] Step 7: Based on the joint supervision mode of multiple downstream tasks such as safety officer abnormal behavior recognition, scene reconstruction, and safety officer face recognition, perform D-level analysis on each decoder. i Training;

[0011] Step 8: Obtain decoding information for downstream tasks such as safety driver abnormal behavior recognition, scene reconstruction, and safety driver face recognition in intelligent driving scenarios. i [M e (S f )).

[0012] Specifically, in step 1, the preprocessing operations such as video dataset collection and video data timestamp alignment, as well as fine-grained manual annotation in the safe driver supervision mode of the intelligent driving scenario, involve fixing the camera inside the vehicle at multiple angles, including the right side of the safe driver and from a top-down view, to collect various action and behavior data of the safe driver in the intelligent driving scenario, including normal driving, hands off the steering wheel, smoking, eating, using a mobile phone, making or receiving calls, disappearing from the camera's view, drinking water, sleeping, etc., forming the original dataset S. raw After performing preprocessing operations such as timestamp alignment, frame rate normalization, scaling and cropping, and segment splitting on the original dataset, a comprehensive fine-grained manually annotated dataset S is formed based on the video data content, targeting various downstream task scenarios.

[0013] Specifically, in step 2, the keyframe selection based on the keyframe filtering algorithm for the input video dataset S is as follows: given the right-side view video data V of a safe driver. a ={v1,v2,v3,...,v n} and top-down video data V b={v′1,v′2,v′3,...,v′ n The keyframe set is filtered using three strategies: selecting a center frame, selecting random frames, and adaptive clustering to select representative frames. For the center frame strategy, the keyframes are selected from V... a and V b A keyframe set is formed by selecting c frames from the center frame and the frames before and after it. and For the random frame strategy, the k-th frame is randomly selected from the video stream as the start frame, resulting in the keyframe set V. Ka ={v k ,v k+1 ,...,v k+c} and V Kb ={v′ k ,v′ k+1 ,...,v′ k+c For adaptive clustering strategies, a pre-trained model is used to perform clustering on V. a and V b Video frame feature extraction was performed to obtain high-dimensional video features F. a ={f1,f2,f3,...,f n} and F b ={f′1,f′2,f′3,...,f′ n Then, an adaptive clustering algorithm is used to combine the high-dimensional features F of video frames from different viewpoints. ab ={(f1,f′1),(f2,f′2),(f3,f′3)...,(f n ,f′ n Clustering is performed to obtain a set of keyframes V that can represent the semantic information of the video data. Ka ={v1,v2,...,v c} and V Kb ={v′1,v′2,...,v′ c}. V Ka and V Kb Together they form the final keyframe set S f Choose an appropriate keyframe filtering strategy based on the needs of downstream business scenarios;

[0014] Specifically, in step 3, the construction of the deep learning-based keyframe feature encoder M for intelligent driving scenarios... e Specifically, a deep convolutional neural network is selected as the encoder M. e The basic components; then select the appropriate number of network layers m and the kernel size parameter K for each layer. i , nuclear number parameter C i and step size parameter B iFurthermore, batch coneification and ReLU activation functions are added after the convolutional layers; then, t convolutional layers are selected to form convolutional blocks, and pooling layers are added between adjacent convolutional blocks for downsampling, selecting appropriate pooling methods and pooling window sizes W. i Window sliding step size H i Finally, a feature encoder network structure that adapts to the semantic information of key frames in video data under intelligent driving scenarios is formed.

[0015] Specifically, in step 4, the keyframe feature encoder M based on the self-supervised intelligent driving scene image reconstruction task... e The training and parameter learning specifically involve: constructing a decoder D2 and an encoder M for image reconstruction tasks in intelligent driving scenarios. e Together they form an image reconstruction network; the keyframe filtering results from step 2, S f The input image serves as both input information and supervisory information for the image reconstruction network. Choosing an appropriate loss function F... loss And gradient optimizer, then train intelligent driving scene image reconstruction network and encoder M through backpropagation e After updating the parameters and the network loss converges, training stops, and the keyframe feature encoder M is obtained. e Parameter weights;

[0016] Specifically, in step 5, the use of feature encoder M e Extract keyframe S f Semantic encoded information M e (S f The data is then uploaded to the blockchain, where multi-node consensus is used to complete the storage and sharing of encoded information. Specifically, this involves uploading key frames of the intelligent driving scenario video data to the blockchain. f As a feature encoder M e The input, the parameter results obtained from training in step 4, are used as the encoder M. e The parameter weights are used to run the model inference to obtain the semantic encoding information M of the input keyframe data. e (S f ); Select an appropriate data uplink method to encode semantic information M e (S f Uploaded to blockchain node N i Multiple consensus nodes on the blockchain run a consensus algorithm, and after reaching a consensus on the currently uploaded data, they encode the semantic information M. e (S f Write the data into the blockchain ledger to complete the notarization and sharing among multiple nodes;

[0017] Specifically, in step 6, the blockchain sharing node N j Using the keyframe semantic encoding information M stored on the chain e (Sf Design downstream task encoder D according to specific business scenarios. i This includes, but is not limited to, the security officer abnormal behavior recognition decoder D1, the scene reconstruction decoder D2, and the security officer face recognition decoder D3, specifically: blockchain shared node N. j Read and parse the semantic encoding information M of the keyframes in the ledger for evidence storage. e (S f Shared node N j Design a safety officer abnormal behavior recognition decoder D1 based on business scenarios (such as the research and testing of autonomous vehicles, and the automatic monitoring of abnormal behavior of safety officers in autonomous logistics); share node N. j Design the intelligent driving scenario reconstruction decoder D2 based on business scenarios (such as traffic accident liability determination, vehicle status monitoring, etc.); share node N. j Design the D3 face recognition decoder for safety officers based on business scenarios (such as verifying authorized drivers, changing supervisors for safety officers, determining and investigating accident liability, etc.);

[0018] Specifically, in step 7, the mode of joint supervision of multiple downstream tasks based on the identification of abnormal behavior of security personnel, scene reconstruction, and facial recognition of security personnel is used to perform D of each decoder. i The training specifically involves: encoding the semantic information M of keyframes in intelligent driving scenario video data. e (S f The labels (or input images) of the corresponding tasks are used as inputs to the downstream task decoders D1, D2, and D3, and the labels of the corresponding tasks are used as supervision information for each decoder; an appropriate loss function L is selected for each downstream task. i And select appropriate weights w i The various loss functions are weighted to form a combined loss function L for collaborative supervision of multiple downstream tasks. mix =∑w i *L i Select an appropriate gradient optimizer to train the multi-downstream task model, and save the D values ​​of each decoder after the network converges. i The weights;

[0019] Specifically, in step 8, the downstream task decoding information D, such as the identification of abnormal behavior of the safety driver, scene reconstruction, and safety driver face recognition in the intelligent driving scenario, is obtained. i [M e (S f Specifically, this involves: encoding the semantic information M of key frames in intelligent driving scenario video data. e (S fThe parameters are input into the safety officer abnormal behavior recognition decoder D1, the scene reconstruction decoder D2, and the safety officer face recognition decoder D3, respectively. The model inference is run using the parameter weights trained in step 7 to obtain the safety officer abnormal behavior recognition result D1[M]. e (S f )]、Intelligent driving scene reconstruction result D2[M e (S f )] and the face recognition result of the security guard D3[M e (S f )). Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0021] Figure 1 Flowchart of the method for semantic encoding on-chain and integration of multiple downstream tasks in intelligent driving scenarios;

[0022] Figure 2 Flowchart for integrating multiple downstream tasks of blockchain shared nodes;

[0023] Figure 3 Flowchart of a method for semantic encoding and on-chaining of intelligent driving scenarios that only include reasoning capabilities, and the integration of multiple downstream tasks. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Current centralized intelligent driving data management solutions have high storage capacity requirements and are susceptible to data loss, tampering, and leakage. They also lack unified data standards and interoperability, resulting in low data utilization efficiency and difficulties in data sharing between different enterprises. This invention addresses these issues by innovatively using deep learning, blockchain, and data encoding / decoding technologies to streamline the overall process framework for feature encoding extraction, on-chain storage and multi-node sharing of massive amounts of data in intelligent driving scenarios, as well as the integration of various downstream tasks. It proposes a method for semantic encoding on-chain and multi-downstream task integration in intelligent driving scenarios, solving the challenges of efficient representation, storage, and sharing of massive and complex data while maintaining data integrity and accuracy. The specific steps are as follows:

[0026] S101: Collect intelligent driving scenario video dataset S raw and the original dataset S raw Preprocessing and manual annotation are performed to form a comprehensive, fine-grained manually annotated dataset S for various downstream task scenarios.

[0027] Specifically, cameras are fixed inside the vehicle at multiple angles, including the right side of the safety driver and from a top-down view, to collect data on various actions and behaviors of the safety driver in intelligent driving scenarios, including but not limited to normal driving, hands off the steering wheel, smoking, eating, using a mobile phone, making or receiving calls, disappearing from the camera's view, drinking water, and sleeping, forming a raw dataset S. raw After performing preprocessing operations such as timestamp alignment, frame rate normalization, scaling and cropping, and segment splitting on the original dataset, a comprehensive fine-grained manually labeled dataset S is formed based on the video data content, which is suitable for various downstream task scenarios such as safety officer abnormal behavior recognition, scene reconstruction, and safety officer face recognition.

[0028] S102: Process the input video data S based on the keyframe filtering algorithm to form a keyframe set S f .

[0029] Specifically, given the video data V from the right-hand perspective of the safe driver a ={v1,v2,v3,...,v n} and top-down video data V b ={v′1,v′2,v′3,...,v′ n The keyframe set is filtered using three strategies: selecting a center frame, selecting random frames, and adaptive clustering to select representative frames. For the center frame strategy, the keyframes are selected from V... a and V b A keyframe set is formed by selecting c frames from the center frame and the frames before and after it. and For the random frame strategy, the k-th frame is randomly selected from the video stream as the start frame, resulting in the keyframe set V. Ka ={v k ,v k+1 ,...,v k+c} and V Kb ={v′ k ,v′ k+1 ,...,v′ k+c For adaptive clustering strategies, a pre-trained model (e.g., CLIP) is used to perform clustering on V. a and V b Video frame feature extraction was performed to obtain high-dimensional video features F. a ={f1,f2,f3,...,f n} and F b ={f′1,f′2,f′3,...,f′ n Then, an adaptive clustering algorithm is used to combine the high-dimensional features of video frames from different perspectives (top-down angle and right-side view of the safety officer) using the F-array algorithm. ab ={(f1,f′1),(f2,f′2),(f3,f′3)...,(f n ,f′ n Clustering is performed to obtain a set of keyframes V that can represent the semantic information of the video data. Ka ={v1,v2,...,v c} and V Kb ={v′1,v′2,...,v′ c}. V Ka and V Kb Together they form the final keyframe set S f Choose an appropriate keyframe filtering strategy based on the needs of downstream business scenarios. For example, for downstream tasks such as identifying abnormal behavior of security personnel, using an adaptive clustering strategy for keyframe filtering has better results.

[0030] S103: Keyframe semantic encoding information of video data available for multiple downstream task scenarios, constructing a deep learning-based keyframe feature encoder M for intelligent driving scenarios. e .

[0031] Specifically, a deep convolutional neural network is chosen as the encoder M. e The basic components; then select the appropriate number of network layers m (e.g., m=6) and the kernel size parameter K for each layer. i , nuclear number parameter C i and step size parameter B iAfter the convolutional layers, batch coneification and ReLU activation functions are added; then, t (e.g., t=2) convolutional layers are selected to form convolutional blocks, and pooling layers are added between adjacent convolutional blocks for downsampling. A suitable pooling method (e.g., max pooling) and pooling window size W are selected. i Window sliding step size H i Finally, a feature encoder network structure that adapts to the semantic information of key frames in video data under intelligent driving scenarios is formed.

[0032] S104: Keyframe Feature Encoder M Based on Self-Supervised Reconstruction Task e Training and parameter learning.

[0033] Specifically, a decoder D2 and an encoder M are constructed for image reconstruction tasks in intelligent driving scenarios. e Together they form an image reconstruction network; the keyframe selection results S f The input image serves as both input information and supervisory information for the image reconstruction network. Choosing an appropriate loss function F... loss (e.g., mean squared error) and gradient optimizer (e.g., Adam), then train the intelligent driving scene image reconstruction network and perform encoder M through backpropagation. e After updating the parameters and the network loss converges, training stops, and the keyframe feature encoder M is obtained. e The parameter weights.

[0034] S105: Obtain semantic encoding information of key frames in video data under intelligent driving scenarios. e (S f The encoded information is then uploaded to the blockchain node, and through multi-node consensus, it is stored and shared.

[0035] Specifically, the key frames S of the intelligent driving scenario video data f As a feature encoder M e The input is the parameter obtained from training, which is used as the encoder M. e The parameter weights are used to run the model inference to obtain the semantic encoding information M of the input keyframe data. e (S f ); Select an appropriate data uploading method (e.g., batch packaging uploading) to encode semantic information M e (S f Uploaded to blockchain node N i Multiple consensus nodes on the blockchain run consensus algorithms (such as identity verification-based consensus algorithms), and after reaching a consensus on the currently uploaded data information, they encode the semantic information M. e (S f The data is written into the blockchain ledger to complete the evidence storage and multi-node sharing.

[0036] S106: Blockchain shared nodes design downstream task encoders D according to specific business scenarios. i .

[0037] Specifically, blockchain shared node N j Read and parse the semantic encoding information M of the keyframes in the ledger for evidence storage. e (S f Shared node N j Design a safety officer abnormal behavior recognition decoder D1 based on business scenarios (such as the research and testing of autonomous vehicles, and the automatic monitoring of abnormal behavior of safety officers in autonomous logistics); share node N. j Design the intelligent driving scenario reconstruction decoder D2 based on business scenarios (such as traffic accident liability determination, vehicle status monitoring, etc.); share node N. j Design the D3 face recognition decoder for safety officers based on business scenarios (such as verifying authorized drivers, changing supervisors for safety officers, determining and investigating accident liability, etc.).

[0038] S107: Based on a multi-downstream task joint supervision model, each decoder performs D... i Training.

[0039] Specifically, the semantic encoding information M of key frames in intelligent driving scenario video data e (S f The labels (or input images) of the corresponding tasks are used as inputs to the downstream task decoders D1, D2, and D3, and the labels of the corresponding tasks are used as supervision information for each decoder; an appropriate loss function L is selected for each downstream task. i And select appropriate weights w i The various loss functions are weighted to form a combined loss function L for collaborative supervision of multiple downstream tasks. mix =∑w i *L i Select an appropriate gradient optimizer (e.g., Adam) to train the multi-downstream task model, and save the D values ​​of each decoder after the network converges. i The weight.

[0040] S108: Obtain decoding information for downstream tasks such as safety driver abnormal behavior recognition, scene reconstruction, and safety driver face recognition in intelligent driving scenarios. i [M e (S f )).

[0041] Specifically, the semantic encoding information M of key frames in intelligent driving scenario video data e (S fThe parameters are input into the safety officer abnormal behavior recognition decoder D1, the scene reconstruction decoder D2, and the safety officer face recognition decoder D3, respectively. The model inference is run using the parameter weights trained in step S107 to obtain the safety officer abnormal behavior recognition result D1[M]. e (S f )]、Intelligent driving scene reconstruction result D2[M e (S f )] and the face recognition result of the security guard D3[M e (S f )).

[0042] For ease of understanding of the present invention, Figure 2 The flowchart illustrates the integration process of various downstream tasks of a blockchain shared node, including but not limited to the construction of decoders for various downstream tasks such as security officer abnormal behavior recognition, scene reconstruction, and security officer facial recognition.

[0043] like Figure 2 The diagram illustrates the integration process of various downstream tasks for a blockchain shared node, specifically including:

[0044] S201: Construct a decoder D1 for recognizing abnormal behavior of safety drivers in intelligent driving scenarios. Specifically, a multilayer perceptron layer is selected as the main part of the decoder D1, using d (e.g., d=3) linear fully connected layers, and an appropriate input size P is selected for each fully connected layer. i and output size Q i In particular, the output size of the last fully connected layer should be consistent with the number of abnormal behavior types. Then, a ReLU activation function is introduced between adjacent fully connected layers to finally form the abnormal behavior recognition decoder D1 for safety drivers in intelligent driving scenarios.

[0045] S202: Construct a scene reconstruction decoder D2 based on the keyframe semantic encoding information on the blockchain. Specifically, a deep deconvolution layer is selected as the basic component of the intelligent driving scene reconstruction task. An appropriate number of deconvolution layers f (e.g., f=3) is selected, and appropriate kernel size and stride parameters are selected for each deconvolution layer. Then, a ReLU activation function is introduced between adjacent deconvolution layers, and finally, the intelligent driving scene reconstruction decoder D2 is formed.

[0046] S203: Construct the safety driver face recognition decoder D3 for intelligent driving scenarios. Specifically, a multilayer perceptron layer is selected as the main body of the safety driver face recognition decoder D3, using d (e.g., d=5) linear fully connected layers, and an appropriate input size P is selected for each fully connected layer. i and output size Q iIn particular, the output size of the last fully connected layer should be consistent with the number of abnormal behavior types. Then, a ReLU activation function is introduced between adjacent fully connected layers to finally form the D3 decoder for recognizing abnormal behavior of the safety driver in the intelligent driving scenario.

[0047] For ease of understanding of the present invention, Figure 3 This diagram illustrates a method for semantic encoding on-chain and multi-downstream task integration in intelligent driving scenarios, which only includes reasoning capabilities. It covers the entire process pathway, including real-time data acquisition and processing, feature encoding on-chain, feature encoding information sharing, and downstream task integration. Specifically, it includes:

[0048] S301: Obtain intelligent driving scenario video dataset S raw and the original dataset S raw Preprocessing operations are performed. Specifically, cameras are fixed inside the vehicle at multiple angles, including the right side of the safety driver and from a top-down view, to collect data on the safety driver's actions and behaviors in intelligent driving scenarios. The raw dataset is then preprocessed with operations such as timestamp alignment, frame rate normalization, scaling and cropping, and segment splitting.

[0049] S302: Process the input video data S based on the keyframe filtering algorithm to obtain the keyframe set S. f Specifically, given the video data V from the right-hand perspective of a safe driver. a ={v1,v2,v3,...,v n} and top-down video data V b ={v′1,v′2,v′3,...,v′ n The keyframe set is filtered using three strategies: selecting a center frame, selecting random frames, and adaptive clustering to select representative frames. For the center frame strategy, the keyframes are selected from V... a and V b A keyframe set is formed by selecting c frames from the center frame and the frames before and after it. and For the random frame strategy, the k-th frame is randomly selected from the video stream as the start frame, resulting in the keyframe set V. Ka ={v k ,v k+1 ,...,v k+c} and V Kb ={v′ k ,v′ k+1 ,...,v′ k+c For adaptive clustering strategies, a pre-trained model is used to perform clustering on V. a and V b Video frame feature extraction was performed to obtain high-dimensional video features F. a ={f1,f2,f3,...,f n} and F b ={f′1,f′2,f′3,...,f′ n Then, an adaptive clustering algorithm is used to combine the high-dimensional features F of video frames from different viewpoints. ab ={(f1,f′1),(f2,f′2),(f3,f′3)...,(f n ,f′ n Clustering is performed to obtain a set of keyframes V that can represent the semantic information of the video data. Ka ={v1,v2,...,v c} and V Kb ={v′1,v′2,...,v′ c}. V Ka and V Kb Together they form the final keyframe set S f Choose an appropriate keyframe filtering strategy based on the needs of downstream business scenarios.

[0050] S303: Set keyframe S f Input Feature Encoder M e Obtain the semantic encoding information M of key frames in video data under intelligent driving scenarios. e (S f This information is then uploaded to blockchain nodes, where multi-node consensus is used to achieve the storage and sharing of coded information. Specifically, the input keyframe set S... f To the feature encoder M e Using the trained encoder weights, the model is run for inference to obtain the semantic encoding information M of keyframes in video data for intelligent driving scenarios. e (S f Then, the feature encoding information M e (S f The information is uploaded to the blockchain node, and the encoded information is stored and shared through multi-node consensus.

[0051] S304: Blockchain shared nodes will encode feature information M e (S f Input to downstream decoder D i The downstream task decoding information D is obtained, including the identification of abnormal behavior of the safety driver, scene reconstruction, and facial recognition of the safety driver in the intelligent driving scenario. i [M e (S f Specifically, blockchain sharing nodes obtain the encoded information M stored on the chain. e (S f After that, based on its actual task requirements, the encoded information M e (S fThe following are inputs to downstream task decoders: D1 for safety driver abnormal behavior recognition, D2 for intelligent driving scene reconstruction, and D3 for safety driver face recognition. Using the trained decoder weights, the model is run to infer the safety driver abnormal behavior recognition decoding information D1[M]. e (S f Intelligent driving scenario reconstruction Jema Information D2[M] e (S f )] and security officer facial recognition decoding information D3[M e (S f )).

[0052] This invention proposes a method for semantic encoding and blockchain integration of intelligent driving scenarios. By semantically encoding and blockchaining massive real-time video data in intelligent driving scenarios, it achieves efficient representation and storage of massive and complex data while maintaining data integrity and accuracy. It also ensures the trustworthiness of the data source while enabling secure data sharing among multiple nodes, meeting the diverse downstream task requirements of blockchain sharing nodes, thereby promoting efficient and trustworthy storage and sharing of data in related fields.

Claims

1. A method for on-chain semantic encoding of intelligent driving scenarios and integration of multiple downstream tasks, characterized in that, The method includes the following steps: Step 1: Collect video dataset S under the safe driver monitoring mode in intelligent driving scenarios. raw For the original dataset S raw After performing preprocessing operations such as timestamp alignment, frame rate normalization, scaling and cropping, and segment splitting, a comprehensive fine-grained manually labeled dataset S is formed based on the video data content, targeting various downstream task scenarios. Step 2: Process the input video data S based on the keyframe selection algorithm, and select keyframes from the video stream S using three methods: center frame strategy, random frame strategy, and adaptive clustering strategy, to form a keyframe set S. f ; Step 3: Based on the semantic encoding information of keyframes in video data available for multiple downstream task scenarios, construct a deep learning-based keyframe feature encoder M for intelligent driving scenarios. e ; Step 4: Perform keyframe feature encoder M based on the self-supervised intelligent driving scene image reconstruction task. e Training and parameter learning; Step 5, use M e Acquiring key frames S of video data in intelligent driving scenarios f Semantic encoded information M e (S f ), and upload it to blockchain node N. i The encoded information is stored and shared through multi-node consensus on the blockchain. Step 6, Blockchain shared node N j Using the keyframe semantic encoding information M stored on the chain e (S f Design the downstream task decoder D according to the specific business scenario. i It includes a security officer abnormal behavior recognition decoder D1, a scene reconstruction decoder D2, and a security officer face recognition decoder D3; Step 7: Based on the joint supervision mode of multiple downstream tasks such as safety officer abnormal behavior recognition, scene reconstruction, and safety officer face recognition, perform D-level analysis on each decoder. i Training; Step 8: Obtain decoding information for downstream tasks such as safety driver abnormal behavior recognition, scene reconstruction, and safety driver face recognition in intelligent driving scenarios. i [M e (S f )).

2. The method for on-chain semantic encoding of intelligent driving scenarios and integration of multiple downstream tasks as described in claim 1, characterized in that, In step 1, the video dataset collection, video data timestamp alignment preprocessing, and fine-grained manual annotation in the intelligent driving scenario under the safe driver supervision mode are specifically performed as follows: A camera is fixed inside the vehicle to the right of the safe driver and from multiple overhead angles to collect various actions and behaviors of the safe driver in the intelligent driving scenario, including normal driving, hands off the steering wheel, smoking, eating, using a mobile phone, making or receiving calls, disappearing from the camera's view, drinking water, and sleeping, forming the original dataset S. raw After performing preprocessing operations such as timestamp alignment, frame rate normalization, scaling and cropping, and segment splitting on the original dataset, a comprehensive fine-grained manually annotated dataset S is formed based on the video data content, catering to various downstream task scenarios.

3. The method for on-chain semantic encoding of intelligent driving scenarios and integration of multiple downstream tasks as described in claim 1, characterized in that, In step 2, the keyframe selection algorithm is used to select keyframes from the input video dataset S. Specifically, given the right-side view video data V of a safe driver... a ={v1,v2,v3,...,v n } and top-down video data V b ={v′1,v′2,v′3,...,v′ n The keyframe set is filtered using three strategies: selecting a center frame, selecting random frames, and adaptive clustering to select representative frames. For the center frame strategy, the keyframes are selected from V... a and V b A keyframe set is formed by selecting c frames from the center frame and the frames before and after it. and For the random frame strategy, the k-th frame is randomly selected from the video stream as the start frame, resulting in a keyframe set V. Ka ={v k ,v k+1 ,...,v k+c } and V Kb ={v′ k ,v′ k+1 ,...,v′ k+c For adaptive clustering strategies, a pre-trained model is used to perform clustering on V. a and V b Video frame feature extraction was performed to obtain high-dimensional video features F. a ={f1,f2,f3,...,f n } and F b ={f′1,f′2,f′3,...,f′ n Then, an adaptive clustering algorithm is used to combine the high-dimensional features F of video frames from different viewpoints. ab ={(f1,f′1),(f2,f′2),(f3,f′3)...,(f n ,f′ n Clustering is performed to obtain a set of keyframes V that can represent the semantic information of the video data. Ka ={v1,v2,...,v c } and V Kb ={v′1,v′2,...,v′ c }, V Ka and V Kb Together they form the final keyframe set S f Choose an appropriate keyframe filtering strategy based on the needs of downstream business scenarios.

4. The method for on-chain semantic encoding of intelligent driving scenarios and integration of multiple downstream tasks as described in claim 1, characterized in that, In step 3, the construction of the deep learning-based keyframe feature encoder M for intelligent driving scenarios is described. e Specifically, a deep convolutional neural network is selected as the encoder M. e The basic components; then select the appropriate number of network layers m and the kernel size parameter K for each layer. i , number of kernels parameter C i and step size parameter B i Furthermore, batch coneification and ReLU activation functions are added after the convolutional layers; then, t convolutional layers are selected to form convolutional blocks, and pooling layers are added between adjacent convolutional blocks for downsampling, selecting appropriate pooling methods and pooling window sizes W. i Window sliding step size H i Finally, a feature encoder network structure that adapts to the semantic information of key frames in video data under intelligent driving scenarios is formed.

5. The method for on-chain semantic encoding of intelligent driving scenarios and integration of multiple downstream tasks as described in claim 1, characterized in that, In step 4, the keyframe feature encoder M, based on the self-supervised intelligent driving scene image reconstruction task, performs... e The training and parameter learning specifically involve: constructing a decoder D2 and an encoder M for image reconstruction tasks in intelligent driving scenarios. e Together they form an image reconstruction network; the keyframe filtering results from step 2, S f The input image serves as both input information and supervisory information for the image reconstruction network. Choosing an appropriate loss function F... loss And gradient optimizer, then train intelligent driving scene image reconstruction network and encoder M through backpropagation e After updating the parameters and the network loss converges, training stops, and the keyframe feature encoder M is obtained. e The parameter weights.

6. The method for on-chain semantic encoding of intelligent driving scenarios and integration of multiple downstream tasks as described in claim 1, characterized in that, In step 5, the feature encoder M is used. e Extract keyframe S f Semantic encoded information M e (S f The data is then uploaded to the blockchain, where multi-node consensus is used to complete the storage and sharing of encoded information. Specifically, this involves uploading key frames of the intelligent driving scenario video data to the blockchain. f As a feature encoder M e The input, the parameter results obtained from training in step 4, are used as the encoder M. e The parameter weights are used to run the model inference to obtain the semantic encoding information M of the input keyframe data. e (S f ); Select an appropriate data uplink method to encode semantic information M e (S f Uploaded to blockchain node N i Multiple consensus nodes on the blockchain run a consensus algorithm, and after reaching a consensus on the currently uploaded data, they encode the semantic information M. e (S f The data is written into the blockchain ledger to complete the evidence storage and multi-node sharing.

7. The method for on-chain semantic encoding of intelligent driving scenarios and integration of multiple downstream tasks as described in claim 1, characterized in that, In step 6, the blockchain sharing node N j Using the keyframe semantic encoding information M stored on the chain e (S f Design the downstream task decoder D according to the specific business scenario. i This includes a security officer abnormal behavior recognition decoder D1, a scene reconstruction decoder D2, and a security officer face recognition decoder D3, specifically: blockchain shared node N. j Read and parse the semantic encoding information M of the keyframes in the ledger for evidence storage. e (S f Shared node N j Design a security officer abnormal behavior recognition decoder D1 based on the business scenario; share node N. j Design intelligent driving scenario reconstruction decoder D2 based on business scenarios; share node N. j The D3 facial recognition decoder for security personnel was designed based on the business scenario.

8. The method for on-chain semantic encoding of intelligent driving scenarios and integration of multiple downstream tasks as described in claim 1, characterized in that, In step 7, the multi-downstream task joint supervision mode based on the identification of abnormal behavior of the safety officer, scene reconstruction, and face recognition of the safety officer is used to perform D of each decoder. i The training specifically involves: encoding the semantic information M of keyframes in intelligent driving scenario video data. e (S f The labels for the corresponding tasks serve as the input to the downstream task decoders D1, D2, and D3, and are used as the supervision information for each decoder. An appropriate loss function L is selected for each downstream task. i And select appropriate weights w i The various loss functions are weighted to form a combined loss function L for collaborative supervision of multiple downstream tasks. mix =∑w i *L i Select an appropriate gradient optimizer to train the multi-downstream task model, and save the D values ​​of each decoder after the network converges. i The weight.

9. The method for on-chain semantic encoding of intelligent driving scenarios and integration of multiple downstream tasks as described in claim 1, characterized in that, In step 8, the downstream task decoding information D for obtaining abnormal behavior recognition of the safety driver, scene reconstruction, and safety driver face recognition in the intelligent driving scenario is... i [M e (S f Specifically, this involves: encoding the semantic information M of key frames in intelligent driving scenario video data. e (S f The parameters are input into the safety officer abnormal behavior recognition decoder D1, the scene reconstruction decoder D2, and the safety officer face recognition decoder D3, respectively. The model inference is run using the parameter weights trained in step 7 to obtain the safety officer abnormal behavior recognition result D1[M]. e (S f )]、Intelligent driving scene reconstruction result D2[M e (S f )] and the face recognition result of the security guard D3[M e (S f )).

Images