New energy automobile automatic driving teaching method based on image recognition interaction feedback

Abstract

The invention discloses a new energy automobile automatic driving teaching method based on image identification interaction feedback. The method comprises the following steps: obtaining a vehicle-mounted multi-view image data set; an HSA-iNeRF model is constructed, semantic consistency multilayer radiation field representation is generated, and frame-level rendering errors are recorded; reverse rendering optimization is executed based on semantic consistency multilayer radiation field representation, and frame-level rendering errors are updated; vehicle operation parameters are collected, and timing constraint attitude feedback optimization is executed; a teacher interaction signal set is collected to generate a modulation weight set, and a reverse rendering optimization process is injected to update semantic consistency multilayer radiation field representation; executing joint optimization, and generating a teaching alignment error matrix and a corrected pose sequence; and generating a teaching feedback instruction set according to the teaching alignment error matrix and the corrected pose sequence. According to the invention, the semantic recognition precision and the feedback adaptive capability in automatic driving teaching are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving systems, and particularly relates to a new energy vehicle automatic driving teaching method based on image recognition interactive feedback. BACKGROUND

[0002] The existing new energy vehicle automatic driving teaching mainly relies on fixed scene recording and rule script driven teaching process, lacks dynamic interactive mechanism based on image recognition, and thus the teaching system is difficult to reflect the vehicle state and environmental changes in real time, and the teacher feedback cannot form an effective closed-loop optimization with the driving model. In addition, the traditional teaching system mostly adopts single-frame recognition or simple multi-camera fusion in vehicle-mounted multi-view image processing, which is difficult to realize spatial consistency and temporal continuity at the semantic level, resulting in accumulated pose estimation error and insufficient teaching scene reproduction accuracy.

[0003] In automatic driving teaching, the existing feedback mechanism based on image recognition is mostly result-level evaluation, which fails to realize the differentiable correlation in the rendering, pose estimation and feedback generation process, and the teacher's operation or voice instruction cannot directly affect the optimization direction of the model, so the real-time and pertinence of teaching interaction are low. At the same time, the existing automatic driving teaching system generally lacks time sequence constraint and feedback modulation structure based on multi-modal input, resulting in learning model update lag, insufficient semantic consistency, and teaching results relying on manual annotation and offline correction, which cannot meet the requirements of real-time feedback and dynamic update in intelligent teaching environment.

[0004] Therefore, how to provide a new energy vehicle automatic driving teaching method based on image recognition interactive feedback is a problem to be solved by those skilled in the art. SUMMARY

[0005] One object of the present application is to provide a new energy vehicle automatic driving teaching method based on image recognition interactive feedback. The present application integrates semantic segmentation, HSA-iNeRF model and differentiable feedback modulation technology, constructs a multi-layer radiation field representation with semantic consistency, and realizes the differentiable correlation between the teacher interactive signal and the vehicle pose optimization. Through the dynamic pose feedback and joint optimization mechanism, the real-time correction of the image recognition model parameters and the pose result is completed, and the teaching guidance result and the feedback instruction set are generated. The present application has the advantages of strong semantic consistency, low feedback delay and high teaching closed-loop adaptability.

[0006] According to the new energy vehicle automatic driving teaching method based on image recognition interactive feedback, the following steps are included:

[0007] Obtain a vehicle-mounted multi-view image data set, input a semantic segmentation network to generate a semantic feature set, and establish a semantic weighting mapping function to generate a semantic saliency weight map;

[0008] The HSA-iNeRF model is constructed, the semantic saliency weight map is loaded into the rendering process, a semantic consistent multi-layer radiance field representation is formed, a rendering error data structure is initialized, and frame-level rendering errors are recorded;

[0009] Based on the semantic consistent multi-layer radiance field representation, the reverse rendering optimization is performed, the initial pose result is generated, and the frame-level rendering error is updated;

[0010] Collecting vehicle operation parameters to generate kinematic prediction, reading initial pose results and frame-level rendering errors to perform timing constraint posture feedback optimization, and outputting dynamic posture feedback sequence;

[0011] Collecting a set of teacher interaction signals to input an interactive feedback encoder to generate a set of modulation weights of a differentiable feedback modulation layer, injecting the reverse rendering optimization process and updating the semantic consistent multi-layer radiance field representation;

[0012] Based on the semantic consistent multi-layer radiance field representation, the dynamic posture feedback sequence and the modulation weight set, the joint optimization is performed, the image recognition model parameters and the pose result are updated, the teaching alignment error matrix and the corrected pose sequence are generated;

[0013] According to the teaching alignment error matrix and the corrected pose sequence, the teaching guide result and the teaching log are generated, and the teaching feedback instruction set directly driving the teaching execution module is output.

[0014] Optionally, the generation of the semantic saliency weight map comprises:

[0015] Collecting a set of vehicle-mounted multi-view image data, performing time synchronization and distortion correction, and writing into an input buffer according to camera number and timestamp;

[0016] Inputting the set of vehicle-mounted multi-view image data in the input buffer into a semantic segmentation network, outputting a set of semantic features and maintaining a one-to-one correspondence with the camera number and the timestamp;

[0017] Based on the semantic feature set, an input index of a semantic weighted mapping function is established at the frame level, and a semantic weight generation rule for each image position is determined;

[0018] Calling the semantic weighted mapping function to generate a semantic saliency weight map, and aligning to the set of vehicle-mounted multi-view image data according to the camera number and the timestamp.

[0019] Optionally, the generation of the semantic consistent multi-layer radiance field representation and the frame-level rendering error comprises:

[0020] The HSA-iNeRF model is constructed, which is composed of a semantic hierarchical coding unit, a semantic guided rendering layer, a volume rendering integration unit and an error measurement unit. The semantic hierarchical coding unit and the semantic guided rendering layer are bidirectionally connected, and the volume rendering integration unit and the error measurement unit are sequentially connected.

[0021] The semantic hierarchical encoding unit receives a semantic feature set, performs multi-layer semantic embedding encoding on the semantic feature set, and outputs a hierarchical semantic feature representation;

[0022] The semantic guidance rendering layer receives a semantic saliency weight map, fuses the semantic weight and the hierarchical semantic feature representation, and generates a semantic weighted ray input structure;

[0023] The volume rendering integration unit performs a weighted volume rendering process according to the semantic weighted ray input structure, calculates radiance values and density values, and generates an initial radiance field feature set;

[0024] The error measurement unit receives the initial radiance field feature set and establishes a rendering error data structure, records frame-level rendering errors, and generates a semantic consistency multi-layer radiance field representation.

[0025] Optionally, the generation of the initial pose result comprises:

[0026] Reading the semantic consistency multi-layer radiance field representation and the rendering error data structure, setting the initialization parameters of the pose parameter interface, establishing a one-to-one correspondence with the camera number and the timestamp;

[0027] Calling the volume rendering integration unit to generate a rendered image under the current parameters of the pose parameter interface, calling the error measurement unit to calculate the pixel-level difference of the corresponding frame and locating the error entry of the frame in the rendering error data structure;

[0028] Performing reverse rendering optimization iteration according to the pixel-level difference and the semantic level information in the rendering error data structure, updating the parameters of the pose parameter interface until the convergence condition is met, and outputting the initial pose result;

[0029] Calling the error measurement unit to recalculate the pixel-level difference of the corresponding frame under the initial pose result, updating the frame-level rendering error in the rendering error data structure, and generating the updated frame-level rendering error.

[0030] Optionally, the generation of the dynamic pose feedback sequence comprises:

[0031] Collecting vehicle CAN bus and IMU data sets, performing time synchronization and zero offset correction, establishing a unified time axis according to the camera number and the timestamp, and generating a preprocessed sensor data set;

[0032] Generating kinematic predictions based on the preprocessed sensor data set through a kinematic state estimation method;

[0033] Reading the initial pose result and the frame-level rendering error in the cache, registering to the kinematic predictions according to the unified time axis and the camera number, and forming a time sequence alignment input set;

[0034] Temporally constrained pose feedback optimization is performed based on the temporally aligned input set. The kinematic prediction is input to the pose parameter interface as the constraint input, and the frame-level rendering error is input to the temporally constrained pose feedback optimization as the weight input, and the dynamic pose feedback sequence is output.

[0035] Optionally, the generation of the modulation weight set includes:

[0036] Collect teacher interaction signals and record alignment information based on camera ID and timestamp;

[0037] The set of teacher interaction signals is input into the interaction feedback encoder to generate a set of interaction feature vectors, and a pixel-level and semantic level alignment index is established with the semantic saliency weight map and hierarchical semantic feature representation.

[0038] Call the differentiable feedback modulation layer, input the set of interactive feature vectors, perform feature weighting mapping on the weight terms of the rendering consistency constraint unit and the error metric unit, and generate a set of modulation weights corresponding to the camera number and timestamp;

[0039] The modulation weight set is injected into the gradient channel of the reverse rendering optimization process, and the proportion of gradient allocation in the semantically saliency weight icon is limited to generate the updated state of the semantically consistent multilayer radiation field representation.

[0040] Write the updated state into the semantically consistent multilayer radiation field representation to generate the updated semantically consistent multilayer radiation field representation.

[0041] Optionally, the generation of the teaching alignment error matrix and the corrected pose sequence includes:

[0042] Read the semantically consistent multilayer radiation field representation, dynamic attitude feedback sequence and modulation weight set, and establish a time-series alignment relationship according to camera number and timestamp to form a joint optimization input set;

[0043] The modulation weight set is input into the weight channel of the joint optimization process, and bound to the corresponding frame and corresponding semantic position in the joint optimization input set as weight input.

[0044] The dynamic attitude feedback sequence is input into the continuity channel of the joint optimization process, and the continuity constraints of adjacent frames are set and a one-to-one correspondence is established with the joint optimization input set.

[0045] Joint optimization iteration is performed based on semantically consistent multilayer radiation field representation. The direction and update amplitude are updated according to the modulation weight set and dynamic pose feedback sequence constraints. The intermediate pose result is generated and the image recognition model parameters are updated to generate the intermediate alignment error matrix.

[0046] The alignment error is recalculated based on the intermediate pose result, the intermediate alignment error matrix is ​​updated, and the convergence condition is determined. When the convergence condition is met, the corrected pose sequence and the teaching alignment error matrix are output.

[0047] Optionally, the generation of the teaching feedback instruction set includes:

[0048] The teaching alignment error matrix and the corrected pose sequence are input into the alignment analysis process. Frame-level alignment and semantic position alignment are performed based on camera number and timestamp to generate an alignment reference set.

[0049] The alignment reference set is input into the guidance generation process. Based on the error magnitude in the teaching alignment error matrix and the attitude change in the corrected pose sequence, the parameters are calculated and prioritized to generate the teaching guidance result.

[0050] Input the alignment reference set and teaching guide results into the record arrangement process, perform temporal arrangement and semantic column division based on timestamp and camera number, and generate teaching log;

[0051] The teaching guidance results are input into the instruction generation process, and parameter encoding and timing encapsulation are performed according to the interface specifications of the teaching execution module to generate a set of teaching feedback instructions.

[0052] The beneficial effects of this invention are:

[0053] First, by introducing semantically consistent multi-layer radiation field representation in the teaching process, we achieved semantically accurate alignment and dynamic rendering of vehicle-mounted multi-view images. This can maintain the consistency of spatial semantic structure and temporal sequence information in complex teaching environments, fundamentally improving the accuracy of image recognition and pose estimation.

[0054] Secondly, by constructing a differentiable feedback modulation layer, this invention enables teacher interaction signals to directly participate in the reverse rendering optimization process, thereby achieving linkage adjustment between teaching feedback and model training. This effectively reduces the reliance on manual intervention and offline annotation, and improves the real-time performance and adaptability of the teaching system.

[0055] Furthermore, this invention introduces a temporally constrained pose optimization structure into the dynamic pose feedback mechanism. By jointly modeling the kinematic prediction and frame-level rendering error, the continuity and stability of pose estimation are enhanced, and the pose drift error between multiple frames is significantly reduced. Attached Figure Description

[0056] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0057] Fig. 1This is an overall flowchart of a teaching method for autonomous driving of new energy vehicles based on image recognition interactive feedback proposed in this invention;

[0058] Fig. 2 This is a schematic diagram of the semantically consistent multilayer radiation field representation construction structure of the HSA-iNeRF model in this invention;

[0059] Fig. 3 This is a schematic diagram illustrating the coupling between the differentiable feedback modulation layer DFML and the reverse rendering optimization process in this invention. Detailed Implementation

[0060] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0061] refer to Figs. 1-3 A teaching method for autonomous driving of new energy vehicles based on image recognition interactive feedback includes the following steps:

[0062] Acquire a set of multi-view in-vehicle images of new energy vehicles in teaching scenarios, input them into a semantic segmentation network to generate a set of semantic features, and simultaneously establish a semantic weighted mapping function to generate a semantic saliency weight map;

[0063] Construct an HSA-iNeRF model, load the semantic saliency weight graph into the rendering process to form a semantically consistent multi-layer radiation field representation, initialize the rendering error data structure and record frame-level rendering errors;

[0064] Inverse rendering optimization is performed based on semantically consistent multi-layer radiation field representation to generate initial pose results and update frame-level rendering error. The updated frame-level rendering error, semantically consistent multi-layer radiation field representation and initial pose results are stored in cache.

[0065] The system collects data from the vehicle's CAN bus and IMU to generate kinematic predictions, reads the initial pose results and frame-level rendering errors from the cache to perform timing constraint pose feedback optimization, outputs a dynamic pose feedback sequence and writes it back to the cache.

[0066] The set of teacher interaction signals is collected and input into the interactive feedback encoder to generate a set of modulation weights for the differentiable feedback modulation layer DFML. The set of modulation weights is then injected into the reverse rendering optimization process and the semantically consistent multilayer radiation field representation in the cache is updated.

[0067] Based on semantically consistent multilayer radiation field representation, dynamic pose feedback sequence and modulation weight set joint optimization is performed to update image recognition model parameters and pose results, and generate teaching alignment error matrix and corrected pose sequence.

[0068] Based on the teaching alignment error matrix and the corrected pose sequence, the teaching guidance results and teaching logs are generated, and a set of teaching feedback instructions that can directly drive the teaching execution module is output.

[0069] In this embodiment, the generation of the semantic saliency weight map includes:

[0070] Collect a set of multi-view images of new energy vehicles in teaching scenarios, perform time synchronization and distortion correction, and write them into the input buffer according to camera number and timestamp;

[0071] The input cached set of in-vehicle multi-view image data is input into the semantic segmentation network, and the output set of semantic features is maintained in a one-to-one correspondence with the camera number and timestamp.

[0072] The semantic segmentation network is specifically the DeepLab V3+ semantic segmentation network. The network uses a dilated convolutional structure to extract multi-scale contextual features. The encoder part consists of a feature extraction module based on the ResNet-101 backbone, which is used to generate multi-scale semantic feature maps. The decoder part is equipped with a cascaded upsampling module and a feature fusion module, which are used to upsample the multi-scale features layer by layer and restore them to the original resolution to output a pixel-by-pixel semantic category map. At the end of the network, a semantic feature extraction layer is set to output a set of semantic features corresponding to the in-vehicle multi-view image data set and maintain a one-to-one correspondence with the camera number and timestamp.

[0073] Based on the semantic feature set, the input index of the semantic weighted mapping function is established at the frame level, the semantic weight generation rule for each image position is determined and written into the weight generation cache;

[0074] The semantic weighted mapping function is called to generate a semantic saliency weight map, which is then aligned with the camera number and timestamp to the vehicle multi-view image dataset and merged into the weight cache.

[0075] The semantic weighted mapping function is specifically a weighted generation function structure composed of a feature response calculation unit, a category weight allocation unit, and a frame-level aggregation unit. The input is the pixel coordinate information corresponding to the semantic feature set and the vehicle multi-view image dataset. The feature response calculation unit is used to extract the channel activation intensity of each pixel in the semantic feature set and normalize it. The category weight allocation unit assigns a fixed weight value to each pixel according to the semantic category confidence. The frame-level aggregation unit maps the semantic weight of each view to the corresponding pixel position and outputs a semantic saliency weight map with the same resolution as the input image. The pixels in the semantic saliency weight map contain the semantic category number and the weight intensity value. Finally, the weight is aligned to the vehicle multi-view image dataset and merged into the weight cache according to the camera number and timestamp order, and used for the construction of the semantically consistent multilayer radiation field representation of the HSA-iNeRF model.

[0076] The semantic feature set and semantic saliency weight map are submitted to the rendering process entry point of the HSA-iNeRF model, where they are associated as inputs and the submission identifier is recorded.

[0077] This step extracts a set of semantic features using the DeepLab V3+ semantic segmentation network and generates a semantic saliency weight map using a semantic weighted mapping function. It transforms the semantic category confidence into pixel-level weighted parameters, thereby realizing explicit constraints on the modeling of the radiation field by semantic information and providing an accurate input foundation for constructing a semantically consistent multilayer radiation field representation for the HSA-iNeRF model.

[0078] In this embodiment, the generation of semantically consistent multilayer radiation field representation and frame-level rendering error includes:

[0079] The HSA-iNeRF model is constructed, consisting of a semantic hierarchical coding unit, a semantic guided rendering layer, a volume rendering integration unit, and an error measurement unit. The semantic hierarchical coding unit and the semantic guided rendering layer are bidirectionally connected, and the volume rendering integration unit and the error measurement unit are sequentially connected.

[0080] The HSA-iNeRF model is short for Hierarchical Semantic Alignment iNeRF model, which is an improved version of the iNeRF model.

[0081] The semantic hierarchical coding unit is used to receive a set of semantic features, perform multi-layer semantic embedding coding on the set of semantic features, and output a hierarchical semantic feature representation;

[0082] The semantically guided rendering layer is used to receive the semantic saliency weight map, fuse the semantic weights with the hierarchical semantic feature representation, generate a semantically weighted ray input structure and pass it to the volume rendering integral unit;

[0083] The volume rendering integral unit performs a weighted volume rendering process based on the semantically weighted ray input structure, calculates the radiance and density values, and generates an initial set of radiance field features.

[0084] The error measurement unit receives the initial radiation field feature set and establishes a rendering error data structure to record frame-level rendering errors and generate a semantically consistent multi-layer radiation field representation. The semantically consistent multi-layer radiation field representation and the rendering error data structure are provided for subsequent steps.

[0085] Specifically, after receiving the initial radiation field feature set, the error measurement unit aligns the data according to the camera number and timestamp, and establishes a hierarchical binding structure with the hierarchical semantic feature representation. After calling the semantic saliency weight map to complete pixel-level weight attachment, the volume rendering integration unit generates the rendered image of the corresponding frame under the initial pose given by the pose parameter interface. The error measurement unit calculates the pixel-by-pixel difference map between the input image and the rendered image of the frame and writes it into the pixel-level index of the rendering error data structure. Then, the pixel-level differences are weighted and summarized according to the semantic level to obtain the semantic level error vector and write it into the semantic level index of the rendering error data structure. On this basis, the weights of all semantic level errors of the frame are summed to obtain a single-value frame-level rendering error and write it into the frame-level index of the rendering error data structure. The rendering error data structure simultaneously holds the pixel-level difference map, the semantic level error vector, and the frame-level rendering error. The error measurement unit outputs the semantically consistent multi-layer radiation field representation accordingly. The semantically consistent multi-layer radiation field representation and the rendering error data structure are provided for subsequent temporal constraint pose feedback optimization and joint optimization.

[0086] This step involves constructing an HSA-iNeRF model and introducing an error metric unit. The semantic saliency weight map is integrated into the volume rendering process to generate a semantically consistent multi-layer radiation field representation and calculate frame-level rendering errors. This achieves deep binding between semantic information and radiation features, forming a multi-layer semantic constraint structure that can be used for pose feedback and optimization.

[0087] In this embodiment, the generation of the initial pose result includes:

[0088] Read the semantically consistent multi-layer radiation field representation and rendering error data structure, set the initialization parameters of the pose parameter interface, and establish a one-to-one correspondence with the camera number and timestamp;

[0089] The volume rendering integration unit is called to generate a rendered image under the current parameters of the pose parameter interface. The error measurement unit is called to calculate the pixel-level difference of the corresponding frame and locate the error entry of the frame in the rendering error data structure.

[0090] Based on the pixel-level differences and semantic level information in the rendering error data structure, perform reverse rendering optimization iteration, update the parameters of the pose parameter interface until the convergence condition is met, and output the initial pose result.

[0091] The error metric unit is invoked to recalculate the pixel-level differences of the corresponding frame under the initial pose result, update the frame-level rendering error in the rendering error data structure, and generate the updated frame-level rendering error.

[0092] The updated frame-level rendering error, semantically consistent multilayer radiation field representation, and initial pose results are written to the cache by camera number and timestamp for subsequent steps to read.

[0093] This step reads the semantically consistent multi-layer radiation field representation and rendering error data structure, optimizes the pose parameter update interface based on reverse rendering, generates an initial pose result that meets the convergence condition, and recalculates the frame-level rendering error under this pose to achieve joint correction of rendering accuracy and pose estimation, providing an accurate initial pose reference for subsequent temporal constraint pose feedback optimization.

[0094] In this embodiment, the generation of the dynamic attitude feedback sequence includes:

[0095] Collect vehicle CAN bus and IMU data sets, perform time synchronization and zero bias correction, establish a unified timeline according to camera number and timestamp, and generate a pre-processed sensor data set;

[0096] Based on the preprocessed sensor data set, kinematic predictions are generated through a kinematic state estimation method. The kinematic state estimation method integrates the linear velocity and direction angle information output from the vehicle's CAN bus and the angular velocity and acceleration information output from the IMU to calculate the vehicle's displacement increment and attitude change under a unified time axis, forming kinematic predictions between consecutive frames, which are used as constraint inputs for subsequent temporal constraint attitude feedback optimization.

[0097] Read the initial pose results and frame-level rendering errors from the cache, register them to the kinematic predictions according to the unified time axis and camera number, and form a temporally aligned input set;

[0098] Based on the temporally aligned input set, temporally constrained pose feedback optimization is performed. The kinematic prediction is input to the pose parameter interface as a constraint input, and the frame-level rendering error is input to the temporally constrained pose feedback optimization as a weight input, and the dynamic pose feedback sequence is output.

[0099] Specifically, firstly, a continuous set of frames on a unified timeline is read, and the initial pose result is aligned frame by frame with the kinematic predictions according to the camera number and timestamp, generating an aligned pose prior sequence and an aligned kinematic prior sequence. Then, based on the frame-level index and pixel-level index of the frame-level rendering error in the rendering error data structure, the error weight vector and pixel difference map for each frame are extracted, and the error weight vector is used to determine the weight allocation for this round of updates. Next, the current parameters of the pose parameter interface are set as the initial pose result, and for each frame on the unified timeline, the following update process is executed sequentially: the kinematic prediction of the frame is read, the deviation between the current pose parameters and the aligned kinematic prior is calculated, and the deviation is weighted and merged with the error weight vector of the frame according to preset weights to form the update target quantity for the frame. Under the constraint of the update target quantity, the volume rendering integral unit is called to generate the rendered image of the frame, and the error metric is called. The unit outputs a new pixel-level difference map and writes back the pixel-level index and frame-level index of the frame in the rendering error data structure. Based on the new pixel-level difference map and the alignment kinematics prior, it calculates the pose increment of the frame, writes the pose increment into the pose parameter interface, and updates the current parameters. Then, for adjacent frames on the same timeline, it calculates the continuity deviation of the pose parameters of adjacent frames, weights and merges the continuity deviation with the frame-level rendering error of adjacent frames to complete a joint correction of the pose parameters of adjacent frames, and synchronously updates the frame-level index value of adjacent frames in the rendering error data structure. Finally, after all frames have completed one traversal update, the above process is repeated according to the same timeline until the decrease in frame-level rendering error of two consecutive traversals is lower than the threshold. The unit outputs a dynamic pose feedback sequence arranged according to the same timeline, and writes the dynamic pose feedback sequence and the update status of the pose parameter interface into the cache for subsequent steps to read.

[0100] This step generates kinematic predictions by fusing vehicle CAN bus and IMU data, and performs temporal constraint attitude feedback optimization based on a time-aligned input set. The kinematic predictions are used as constraint inputs and frame-level rendering errors are used as weight inputs. The pose parameter interface is iteratively updated to generate a dynamic attitude feedback sequence, achieving collaborative optimization of multi-frame pose continuity and minimization of rendering errors.

[0101] In this embodiment, the generation of the modulation weight set includes:

[0102] Collect a set of teacher interaction signals, which includes image clicks, box selection and dragging, and voice command keywords, and record alignment information based on camera number and timestamp;

[0103] The set of teacher interaction signals is input into the interaction feedback encoder to generate a set of interaction feature vectors, and a pixel-level and semantic level alignment index is established with the semantic saliency weight map and hierarchical semantic feature representation.

[0104] The interactive feedback encoder is specifically a multimodal feature coding structure composed of a signal parsing unit, a temporal coding unit, a semantic alignment unit, and a feature fusion unit connected sequentially. The signal parsing unit receives image click, box selection and drag, and voice command keyword signals from the teacher's interactive signal set and converts different types of signals into a unified feature representation. The temporal coding unit performs positional encoding and temporal series modeling on the parsed feature sequence based on timestamp information to maintain the temporal correlation of interactive signals between consecutive frames. The semantic alignment unit calls the semantic saliency weight map and hierarchical semantic feature representation to perform semantic position mapping on the temporal features and establish the alignment relationship between interactive signals and semantic levels. The feature fusion unit performs weighted fusion of the mapped temporal features and semantic embedded features to generate an interactive feature vector set. Each feature vector in the interactive feature vector set corresponds to the camera number, timestamp, and semantic category number, which is used by the differentiable feedback modulation layer (DFML) to generate a modulation weight set.

[0105] The Differentiable Feedback Modulation Layer (DFML) is invoked, and the set of interactive feature vectors is input into the DFML. The DFML is composed of a channel weighting layer, a gradient gating unit, and a feature fusion layer connected sequentially. The channel weighting layer is used to allocate initial weights according to the channel response intensity of the set of interactive feature vectors. The gradient gating unit controls the opening and closing state of the gradient channels based on the semantic saliency weight map and the dynamic pose feedback sequence, limiting the gradient propagation range. The feature fusion layer, while maintaining the correspondence between camera number and timestamp, maps the features after channel weighting and gradient gating to the weight terms of the rendering consistency constraint unit and the error metric unit, performs feature weighting mapping, and generates a set of modulation weights corresponding to the camera number and timestamp. The DFML is inserted in the volume rendering integral unit sampling weight update stage of the HSA-iNeRF model. It achieves interactive guidance by differentiably modulating the volume rendering gradient path. During its optimization process, it simultaneously minimizes the weighted combination of the modulation loss function and the main rendering loss function. The modulation loss function is used to constrain the influence of interactive features on the gradient direction and magnitude to ensure the stable convergence and dynamic controllable update of the semantically consistent multilayer radiation field representation.

[0106] The modulation weight set is injected into the gradient channel of the reverse rendering optimization process, and the proportion of gradient allocation in the semantically saliency weight icon is limited to generate the updated state of the semantically consistent multilayer radiation field representation.

[0107] The updated state is written into the semantically consistent multilayer radiation field representation, generating the updated semantically consistent multilayer radiation field representation. The modulation weight set and the updated semantically consistent multilayer radiation field representation are then written into the cache according to the camera number and timestamp for subsequent steps to read.

[0108] This step parses the teacher's interaction signals through an interactive feedback encoder, generates a set of interactive feature vectors corresponding to the semantic level, performs feature weighting mapping in the differentiable feedback modulation layer DFML to form a set of modulation weights, injects this set into the reverse rendering optimization process to achieve gradient allocation modulation, updates the semantically consistent multilayer radiation field representation, and completes the dynamic control of interactive feedback in model optimization.

[0109] In this embodiment, the generation of the teaching alignment error matrix and the corrected pose sequence includes:

[0110] Read the semantically consistent multilayer radiation field representation, dynamic attitude feedback sequence and modulation weight set, and establish a time-series alignment relationship according to camera number and timestamp to form a joint optimization input set;

[0111] The modulation weight set is input into the weight channel of the joint optimization process, and bound to the corresponding frame and corresponding semantic position in the joint optimization input set as weight input.

[0112] The dynamic attitude feedback sequence is input into the continuity channel of the joint optimization process, and the continuity constraints of adjacent frames are set and a one-to-one correspondence is established with the joint optimization input set.

[0113] Joint optimization iteration is performed based on semantically consistent multilayer radiation field representation. The direction and update amplitude are updated according to the modulation weight set and dynamic pose feedback sequence constraints. The intermediate pose result is generated and the image recognition model parameters are updated to generate the intermediate alignment error matrix.

[0114] Recalculate the alignment error under the intermediate pose result, update the intermediate alignment error matrix and determine the convergence condition. When the convergence condition is met, output the corrected pose sequence and the teaching alignment error matrix.

[0115] The teaching alignment error matrix and the corrected pose sequence are written into the cache according to the camera number and timestamp, and then used for subsequent steps.

[0116] This step integrates semantically consistent multilayer radiation field representation, dynamic pose feedback sequence, and modulation weight set to perform joint optimization iteration. Based on semantic constraints and continuity constraints, it synchronously updates the image recognition model parameters and pose results, generating a teaching alignment error matrix and a corrected pose sequence, thereby achieving a synchronous improvement in model recognition accuracy and pose estimation consistency during the teaching process.

[0117] In this embodiment, the generation of the teaching feedback instruction set includes:

[0118] The teaching alignment error matrix and the corrected pose sequence are input into the alignment analysis process. Frame-level alignment and semantic position alignment are performed based on camera number and timestamp to generate an alignment reference set.

[0119] The alignment reference set is input into the guidance generation process. Based on the error magnitude in the teaching alignment error matrix and the attitude change in the corrected pose sequence, the parameters are calculated and prioritized to generate the teaching guidance result. The teaching guidance result includes operation correction parameters and visual prompt parameters.

[0120] Input the alignment reference set and teaching guide results into the record arrangement process, perform temporal arrangement and semantic column division based on timestamp and camera number, and generate teaching log;

[0121] The teaching guidance results are input into the instruction generation process, and parameter encoding and timing encapsulation are performed according to the interface specifications of the teaching execution module to generate a set of teaching feedback instructions;

[0122] The teaching guidance results, teaching logs, and teaching feedback instruction sets are written to the cache according to the camera number and timestamp, and are used by the teaching execution module for reading and execution.

[0123] This step generates teaching guidance results that include operation corrections and visual prompts by performing alignment analysis and parameter calculation on the teaching alignment error matrix and the corrected pose sequence. It also performs coding and encapsulation according to the interface specification to form a set of teaching feedback instructions, thereby realizing the structured output of teaching content and the automatic driving linkage of the teaching execution module.

[0124] Example 1:

[0125] To verify the feasibility of this invention in practice, it was applied to a teaching system for autonomous driving in new energy vehicles to test its semantic recognition accuracy, attitude estimation stability, and interactive feedback response efficiency in complex driving scenarios. This scenario includes three teaching environments: urban roads, a closed test area, and a virtual driving simulator. The system constructs a closed-loop teaching interaction system using multi-view vehicle cameras, a CAN bus, an IMU module, and a teacher interaction terminal for visual recognition, attitude correction, and teaching feedback generation during autonomous driving teaching.

[0126] In practical applications, the vehicle-mounted camera acquires multi-view image data at a resolution of 1920×1080 at a frequency of 30 frames per second. The IMU output frequency is 200Hz, and the CAN bus data sampling period is 10ms, ensuring synchronous acquisition of images and sensor signals. The system first uses the DeepLab V3+ semantic segmentation network to generate a semantic feature set and then constructs a semantic saliency weight map through a semantic weighted mapping function. Taking 50 teaching scenarios as an example, the system acquires approximately 120,000 frames of image data, generating semantic feature vectors of approximately 3.6×10⁻⁶. 8 The HSA-iNeRF model forms a semantically consistent multilayer radiation field representation after loading a semantic saliency weight map. It outputs frame-level rendering error through volume rendering integration and error measurement processes, which is used for subsequent pose correction and teacher feedback learning.

[0127] In the interactive teaching phase, teachers dynamically intervene in the student's driving process using voice commands and a visual annotation terminal, including target object recognition and correction, path adjustment, and driving behavior evaluation. The interactive signals are parsed by the interactive feedback encoder and converted into semantic-level interactive feature vectors, which are then input to the differentiable feedback modulation layer (DFML). The DFML module performs dynamic weight modulation on the gradient channels during the reverse rendering optimization process, enabling the system to automatically adjust semantic error weights based on teacher feedback during model training, thereby improving the coupling and adaptive capability of semantic recognition and pose estimation. After actual operation testing, the system achieves an average feedback latency of less than 0.12 seconds in continuous teaching tasks.

[0128] In the attitude estimation and correction stage, the system generates kinematic predictions by fusing data from the CAN bus and IMU, and uses these predictions as temporal constraints input to the attitude feedback optimization module. The system exhibits high stability in attitude drift control across consecutive frames, achieving an average attitude error of 0.37° on the test set, a reduction of approximately 42% compared to traditional NeRF-based methods without constraint optimization. Simultaneously, the semantically consistent multilayer radiation field representation demonstrates stable illumination consistency and semantic boundary alignment performance across 50 teaching scenarios, reducing the average pixel-level rendering error to 0.0068 and improving the inter-frame semantic consistency score by approximately 26%.

[0129] After comprehensive training and multiple iterations of optimization, this system has achieved the functions of automatically identifying erroneous driving behaviors, generating posture correction instructions, and producing teaching logs during the teaching process. Teacher interaction signals are dynamically modulated using DFML, enabling the system to inject precise feedback based on semantic hierarchical features, thereby forming a real-time set of teaching feedback instructions to drive the teaching execution module for visual teaching guidance. The system supports automatic generation of teaching logs and semantically segmented recording, with an average log generation time of only 0.4 seconds, significantly improving the automation level of teaching management.

[0130] The table below shows the performance comparison results of this invention and traditional autonomous driving teaching methods. The test indicators cover dimensions such as semantic recognition accuracy, pose estimation error, feedback response time, and teaching convergence cycle.

[0131] Table 1. Performance Comparison of the Invention and Traditional Methods

[0132] Test metrics Conventional NeRF-teaching system Method of the invention Semantic segmentation mean intersection over union (mIoU) 81.6% 89.8% Mean pose estimation error (°) 0.64 0.37 Inter-frame rendering consistency score (0-1) 0.73 0.92 Teacher feedback response latency (seconds) 0.28 0.12 Average teaching task convergence period (times) 18 9 Log generation time consumption (seconds) 1.6 0.4 Semantic error recovery rate (%) 68.2 88.5 Overall performance improvement amplitude (%) — Approximately 38.7

[0133] Experimental results show that this invention outperforms traditional methods in several key metrics. The average intersection-union ratio (IUU) of semantic segmentation is improved by 8.2%, indicating that the semantically consistent multilayer radiation field representation significantly enhances the semantic boundary recognition capability in complex scenes. The pose estimation error is reduced by 42%, demonstrating that the introduction of temporally constrained pose feedback optimization effectively mitigates the cumulative effect of multi-frame drift. The feedback response delay is shortened by 57%, reflecting the efficient dynamic adaptation mechanism of the differentiable feedback modulation layer in teacher interaction feedback. The convergence cycle of teaching tasks is shortened by 50%, showing the system's convergence efficiency in adaptive training. The semantic error recovery rate is improved by 20.3%, verifying the positive regulatory effect of interactive signals on model optimization. This invention, through multimodal fusion and differentiable feedback mechanisms, achieves real-time closed-loop optimization of image recognition, pose estimation, and teaching feedback, demonstrating significant intelligent and high-precision advantages in autonomous driving teaching systems.

[0134] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A teaching method for autonomous driving of new energy vehicles based on image recognition interactive feedback, characterized in that, Includes the following steps: Acquire a set of multi-view in-vehicle image data, input it into a semantic segmentation network to generate a set of semantic features, and at the same time establish a semantic weighted mapping function to generate a semantic saliency weight map; Construct an HSA-iNeRF model, load the semantic saliency weight graph into the rendering process to form a semantically consistent multi-layer radiation field representation, initialize the rendering error data structure and record frame-level rendering errors; Based on semantically consistent multi-layer radiation field representation, reverse rendering optimization is performed to generate initial pose results and update frame-level rendering errors. Collect vehicle operating parameters to generate kinematic predictions, read initial pose results and frame-level rendering errors to perform temporal constraint pose feedback optimization, and output dynamic pose feedback sequence. The set of teacher interaction signals is collected and input into the interactive feedback encoder to generate a set of modulation weights for the differentiable feedback modulation layer. This set is then injected into the reverse rendering optimization process and the semantically consistent multilayer radiation field representation is updated. Based on semantically consistent multilayer radiation field representation, dynamic pose feedback sequence and modulation weight set joint optimization is performed to update image recognition model parameters and pose results, and generate teaching alignment error matrix and corrected pose sequence. Based on the teaching alignment error matrix and the corrected pose sequence, the teaching guidance results and teaching logs are generated, and a set of teaching feedback instructions that can directly drive the teaching execution module is output.

2. The teaching method for autonomous driving of new energy vehicles based on image recognition interactive feedback according to claim 1, characterized in that, The generation of the semantic saliency weight map includes: Collect a set of multi-view image data from the vehicle, perform time synchronization and distortion correction, and write the data into the input buffer according to the camera number and timestamp. The input cached set of in-vehicle multi-view image data is input into the semantic segmentation network, and the output set of semantic features is maintained in a one-to-one correspondence with the camera number and timestamp. Based on the semantic feature set, the input index of the semantic weighted mapping function is established at the frame level to determine the semantic weight generation rule for each image location; The semantic weighted mapping function is called to generate a semantic saliency weight map, which is then aligned to the vehicle-mounted multi-view image dataset by camera number and timestamp.

3. The teaching method for autonomous driving of new energy vehicles based on image recognition interactive feedback according to claim 1, characterized in that, The semantically consistent multilayer radiation field representation and frame-level rendering error generation include: The HSA-iNeRF model is constructed, consisting of a semantic hierarchical coding unit, a semantic guided rendering layer, a volume rendering integration unit, and an error measurement unit. The semantic hierarchical coding unit and the semantic guided rendering layer are bidirectionally connected, and the volume rendering integration unit and the error measurement unit are sequentially connected. The semantic hierarchical coding unit receives a set of semantic features, performs multi-layer semantic embedding coding on the set of semantic features, and outputs a hierarchical semantic feature representation; The semantically guided rendering layer receives the semantic saliency weight map, fuses the semantic weights with the hierarchical semantic feature representations, and generates a semantically weighted ray input structure. The volume rendering integral unit performs a weighted volume rendering process based on the semantically weighted ray input structure, calculates the radiance and density values, and generates an initial set of radiance field features. The error measurement unit receives the initial radiation field feature set and establishes a rendering error data structure, records frame-level rendering errors, and generates a semantically consistent multi-layer radiation field representation.

4. The teaching method for autonomous driving of new energy vehicles based on image recognition interactive feedback according to claim 1, characterized in that, The generation of the initial pose result includes: Read the semantically consistent multi-layer radiation field representation and rendering error data structure, set the initialization parameters of the pose parameter interface, and establish a one-to-one correspondence with the camera number and timestamp; The volume rendering integration unit is called to generate a rendered image under the current parameters of the pose parameter interface. The error measurement unit is called to calculate the pixel-level difference of the corresponding frame and locate the error entry of the frame in the rendering error data structure. Based on the pixel-level differences and semantic level information in the rendering error data structure, perform reverse rendering optimization iteration, update the parameters of the pose parameter interface until the convergence condition is met, and output the initial pose result. The error metric unit is invoked to recalculate the pixel-level differences of the corresponding frame under the initial pose result, update the frame-level rendering error in the rendering error data structure, and generate the updated frame-level rendering error.

5. The teaching method for autonomous driving of new energy vehicles based on image recognition interactive feedback according to claim 1, characterized in that, The generation of the dynamic attitude feedback sequence includes: Collect vehicle CAN bus and IMU data sets, perform time synchronization and zero bias correction, establish a unified timeline according to camera number and timestamp, and generate a pre-processed sensor data set; Kinematic predictions are generated based on preprocessed sensor data sets using kinematic state estimation methods. Read the initial pose results and frame-level rendering errors from the cache, register them to the kinematic predictions according to the unified time axis and camera number, and form a temporally aligned input set; Temporally constrained pose feedback optimization is performed based on the temporally aligned input set. The kinematic prediction is input to the pose parameter interface as the constraint input, and the frame-level rendering error is input to the temporally constrained pose feedback optimization as the weight input, and the dynamic pose feedback sequence is output.

6. The teaching method for autonomous driving of new energy vehicles based on image recognition interactive feedback according to claim 1, characterized in that, The generation of the modulation weight set includes: Collect teacher interaction signals and record alignment information based on camera ID and timestamp; The set of teacher interaction signals is input into the interaction feedback encoder to generate a set of interaction feature vectors, and a pixel-level and semantic level alignment index is established with the semantic saliency weight map and hierarchical semantic feature representation. Call the differentiable feedback modulation layer, input the set of interactive feature vectors, perform feature weighting mapping on the weight terms of the rendering consistency constraint unit and the error metric unit, and generate a set of modulation weights corresponding to the camera number and timestamp; The modulation weight set is injected into the gradient channel of the reverse rendering optimization process, and the proportion of gradient allocation in the semantically saliency weight icon is limited to generate the updated state of the semantically consistent multilayer radiation field representation. Write the updated state into the semantically consistent multilayer radiation field representation to generate the updated semantically consistent multilayer radiation field representation.

7. The teaching method for autonomous driving of new energy vehicles based on image recognition interactive feedback according to claim 1, characterized in that, The generation of the teaching alignment error matrix and the corrected pose sequence includes: Read the semantically consistent multilayer radiation field representation, dynamic attitude feedback sequence and modulation weight set, and establish a time-series alignment relationship according to camera number and timestamp to form a joint optimization input set; The modulation weight set is input into the weight channel of the joint optimization process, and bound to the corresponding frame and corresponding semantic position in the joint optimization input set as weight input. The dynamic attitude feedback sequence is input into the continuity channel of the joint optimization process, and the continuity constraints of adjacent frames are set and a one-to-one correspondence is established with the joint optimization input set. Joint optimization iteration is performed based on semantically consistent multilayer radiation field representation. The direction and update amplitude are updated according to the modulation weight set and dynamic pose feedback sequence constraints. The intermediate pose result is generated and the image recognition model parameters are updated to generate the intermediate alignment error matrix. The alignment error is recalculated based on the intermediate pose result, the intermediate alignment error matrix is ​​updated, and the convergence condition is determined. When the convergence condition is met, the corrected pose sequence and the teaching alignment error matrix are output.

8. The teaching method for autonomous driving of new energy vehicles based on image recognition interactive feedback according to claim 1, characterized in that, The generation of the teaching feedback instruction set includes: The teaching alignment error matrix and the corrected pose sequence are input into the alignment analysis process. Frame-level alignment and semantic position alignment are performed based on camera number and timestamp to generate an alignment reference set. The alignment reference set is input into the guidance generation process. Based on the error magnitude in the teaching alignment error matrix and the attitude change in the corrected pose sequence, the parameters are calculated and prioritized to generate the teaching guidance result. Input the alignment reference set and teaching guide results into the record arrangement process, perform temporal arrangement and semantic column division based on timestamp and camera number, and generate teaching log; The teaching guidance results are input into the instruction generation process, and parameter encoding and timing encapsulation are performed according to the interface specifications of the teaching execution module to generate a set of teaching feedback instructions.

Images