Scene label generation method and device based on dynamic graph reasoning

Abstract

The invention provides a scene label generation method and device based on dynamic graph reasoning, and relates to the field of automatic driving. The method comprises the following steps: performing open set target detection on a collected environment image based on a visual language large model to obtain a target detection result, and generating an atomic-scale structured tag according to the target detection result; constructing a space-time sequence relation graph based on the atomic-scale structured labels of the continuous frames, and carrying out dynamic interaction modeling on the graph by adopting a graph neural network; and performing semantic reasoning on the space-time sequence relation graph in combination with a traffic rule knowledge base, and outputting a scene-level semantic tag as a final scene understanding result. The method solves the problems that traditional artificial scene labeling is low in efficiency and high in cost, and an existing deep learning method lacks fine-grained label generation and semantic reasoning ability and limits the scene understanding depth, and is mainly used for improving the perception ability and decision-making level of an automatic driving system in a complex environment.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving, and in particular to a method and device for generating scene labels based on dynamic graph reasoning. Background Technology

[0002] With the rapid development of autonomous driving technology, its application scenarios have expanded from single urban roads to various environments such as highways, rural roads, and complex weather conditions. Traditional scene annotation methods mainly rely on manual work, which suffers from low efficiency, high cost, and susceptibility to subjective factors, resulting in inconsistent annotation quality and failing to meet the needs of autonomous driving systems for large-scale, high-quality scene data. Although deep learning-based automated annotation methods have alleviated this problem to some extent, how to achieve efficient and accurate scene understanding remains a pressing technical challenge in this field.

[0003] In existing technologies, related methods attempt to label autonomous driving datasets using multimodal large models. Specifically, this involves initializing and setting the parameters of the multimodal large model using natural language commands, automatically labeling the input autonomous driving dataset using the model, and optimizing the labeling results using a multi-stage quality verification mechanism. This approach focuses on improving the quality management of the labeling process, ensuring labeling accuracy through repeated quality verification.

[0004] However, existing technologies still have significant shortcomings: related methods mainly focus on the quality control process of data annotation, lacking the ability to generate fine-grained labels using large visual models and combine them with semantic analysis to mine deeper scene information. In particular, they fail to address the key technical aspects of constructing scene graphs by analyzing the semantic relationships between targets, and generating high-level scene labels based on graph reasoning, which limits the depth and practicality of scene understanding. Summary of the Invention

[0005] This invention aims to at least solve the technical problems existing in the prior art. To this end, the first aspect of this invention proposes a scene label generation method based on dynamic graph reasoning, the method comprising:

[0006] Open set target detection is performed on the collected environmental images based on a large visual language model to obtain target detection results, and atomic-level structured labels are generated based on the target detection results;

[0007] A spatial-temporal relational graph is constructed based on the atomic-level structured tags of consecutive frames, and a graph neural network is used to perform dynamic interactive modeling of the graph;

[0008] The spatial-temporal relationship graph is semantically reasoned using a traffic rules knowledge base, and scene-level semantic labels are output as the final scene understanding result.

[0009] Optionally, before performing open-set target detection on the acquired environmental images based on the large visual language model, the method further includes:

[0010] The acquired environmental images are preprocessed, and the preprocessing includes at least one of the following operations: image denoising, resolution normalization, lens distortion correction, and color space conversion.

[0011] Optionally, the generation of atomic-level structured tags includes:

[0012] Based on a multi-task learning framework, the detection results of the open set object detection are used to perform object detection, attribute prediction and geometric estimation, and output atomic-level structured labels;

[0013] The atomic-level structured label includes target category label, spatial location information, physical size parameters, and semantic attribute description.

[0014] Optionally, the construction of the spatial-temporal relationship map includes:

[0015] Using the targets in the target detection results as nodes, the atomic-level structured labels as node attributes, and the spatial relative relationships, cross-frame temporal associations, and dynamic interaction modes between targets as edge relationships, a spatial-temporal relationship graph is constructed.

[0016] Optionally, the process of constructing the spatial-temporal relationship map includes:

[0017] The system performs association matching on the same target in multiple consecutive frames to establish cross-frame target correspondence; it performs smoothing processing and trajectory prediction on the motion state of the associated target to obtain continuous motion information of the target; and it dynamically updates the map structure based on the association and prediction results to form a spatial-temporal relationship map that reflects the dynamic changes of the scene.

[0018] Optionally, the step of using a graph neural network to perform dynamic interaction modeling of the graph includes:

[0019] By employing graph convolutional networks or graph attention networks, and aggregating the neighbor node information of each node through a multi-layer message passing mechanism, the contextual representation of each target is learned.

[0020] Optionally, the method further includes:

[0021] The environmental image is input into the illumination condition analysis module, and the illumination analysis results are output.

[0022] The confidence level of the atomically structured tag is adaptively calibrated based on the illumination analysis results and target detection results.

[0023] Optionally, the method further includes:

[0024] A dual-channel architecture is adopted to store scene tags. The real-time channel stores the generated scene-level semantic tags in a high-speed cache, while the archive channel uses feature distillation to compress and store the scene-level semantic tags and feature vectors.

[0025] A second aspect of the present invention provides an electronic device comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or at least one program being loaded and executed by the processor to implement the scene tag generation method based on dynamic graph reasoning as described in the first aspect.

[0026] The beneficial effects of a scene label generation method and device based on dynamic graph reasoning are:

[0027] By linking three modules—open set object detection, spatial-temporal relation graph construction, and semantic reasoning—to form an end-to-end hierarchical scene understanding pipeline, a leap from pixel-level information to atomic-level object perception and then to scene-level semantic understanding is achieved. A dynamic label confidence correction mechanism based on illumination adaptation analyzes illumination conditions in real time and dynamically adjusts detection parameters, effectively improving perception stability and reliability under different illumination environments. A dual-track parallel storage architecture of "real-time channel + archive channel," combined with feature distillation and compression algorithms, enables efficient storage and rapid retrieval of scene label data. The various technical modules work together to form a complete technical system, jointly enhancing the perception capabilities and decision-making level of the autonomous driving system in complex environments. Attached Figure Description

[0028] Figure 1 A flowchart illustrating a scene tag generation method based on dynamic graph reasoning, provided in an embodiment of the present invention;

[0029] Figure 2 This is a flowchart illustrating the atomic-level structured tag generation process in an embodiment of the present invention;

[0030] Figure 3 This is a schematic diagram of the spatial-temporal relationship graph in an embodiment of the present invention;

[0031] Figure 4 This is a schematic diagram of the semantic reasoning process in an embodiment of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more. Furthermore, the use of "based on" or "according to" implies openness and inclusiveness, because processes, steps, calculations, or other actions "based on" or "according to" one or more of the stated conditions or values ​​may in practice be based on additional conditions or beyond the stated values.

[0034] This invention provides a method for generating scene tags based on dynamic graph reasoning, such as... Figure 1 As shown, the method may include the following steps:

[0035] Step 101: Perform open set target detection on the acquired environmental images based on the visual language large model to obtain the target detection results, and generate atomic-level structured labels based on the target detection results.

[0036] Specifically, the environmental images are acquired through the vehicle's onboard vision sensors, which obtain image data of the driving environment. The image data includes external environmental elements such as roads, traffic signals, vehicles, and pedestrians. The types of image data acquired cover different weather conditions, lighting changes (day / night), and complex traffic scenarios to ensure the ability to handle various real-world driving situations.

[0037] Specifically, such as Figure 2 As shown, the input of the visual language big model includes a text cue library. Based on the cross-modal alignment mechanism, the image features of the environment image and the text features in the text cue library are embedded into the same high-dimensional space for similarity calculation. By encoding the image into blocks and using the self-attention mechanism to fuse visual and text information, the classification and localization of the target are achieved, thus completing the target detection.

[0038] Large-scale visual language models (such as Grounding DINO) possess zero-shot and open-set recognition capabilities, enabling the identification and localization of various targets in images. They not only cover predefined categories of common traffic participants such as vehicles, pedestrians, and traffic signs, but also identify emerging target categories not seen during training, such as special types of engineering vehicles, temporary obstacles, or new traffic facilities. Grounding DINO is an end-to-end visual language detection model that employs dual encoders (image Swing Transformer + text BERT) and a cross-modal decoder. Through language-guided feature selection and text-image fusion attention, it achieves open-vocabulary target detection and supports direct object localization from natural language input without the need for predefined categories. It performs exceptionally well on zero-shot detection tasks, effectively ensuring the comprehensiveness and accuracy of open-set target detection.

[0039] In one possible implementation, before performing open set target detection on the acquired environmental images based on the visual language large model, the acquired environmental images are further preprocessed. The preprocessing includes at least one of the following operations: image denoising, resolution normalization, lens distortion correction, and color space conversion.

[0040] Specifically, to ensure the accuracy of subsequent target detection and label generation, the acquired environmental images undergo preprocessing operations, including but not limited to:

[0041] Denoising: Remove noise from the image by median filtering, Gaussian filtering or other noise reduction techniques to improve image quality and avoid noise interfering with target detection;

[0042] Resolution adjustment: Unify the image resolution to a standard size suitable for model processing (such as 640×480 or 1920×1080), achieving a balance between preserving image details and meeting computational resource and real-time requirements;

[0043] Distortion correction: Based on camera calibration parameters, lens distortion (such as fisheye distortion and perspective distortion) is corrected to ensure the accurate shape and proportion of objects in the image and avoid target recognition errors caused by distortion.

[0044] Color space conversion: Based on the characteristics of different sensors, the image is converted from one color space (such as RGB) to another color space (such as YUV or grayscale image) to better adapt to the input requirements of subsequent processes.

[0045] In one possible implementation, generating atomic-level structured labels includes: performing target detection, attribute prediction, and geometric estimation on the detection results of open set target detection based on a multi-task learning framework, and outputting atomic-level structured labels; the atomic-level structured labels include target category labels, spatial location information, physical size parameters, and semantic attribute descriptions.

[0046] Specifically, atomic-level labels are highly fine-grained semantic descriptions containing multi-dimensional attribute information: target category labels include the major category to which it belongs and its subcategories, such as a three-level classification system of "vehicle-truck-refrigerated truck"; spatial location information is represented by pixel coordinate bounding boxes and 3D position estimation in the camera coordinate system; physical size parameters include physical measurements such as length, width, and height, estimated based on monocular or binocular visual perception technology combined with prior knowledge; semantic attribute descriptions cover auxiliary semantic features such as color, motion state, vehicle model, and pedestrian posture. This multi-dimensional structured representation provides a rich semantic foundation for subsequent scene understanding and semantic reasoning.

[0047] Specifically, the multi-task learning framework simultaneously performs object detection, attribute prediction, and geometric estimation. For example, for a detected vehicle, it not only outputs its position but also, through an attribute prediction branch, outputs label information such as its color, estimated speed, and vehicle type. The target information obtained by the multi-task learning framework is organized in a structured form and serves as the basic input for subsequent processes.

[0048] Step 102: Construct a spatial-temporal relationship graph based on atomic-level structured tags of continuous frames, and use graph neural networks to perform dynamic interaction modeling on the graph to generate dynamic interaction relationships.

[0049] Specifically, the construction process of the spatial-temporal relationship graph includes three main steps: data association, trajectory prediction, and graph structure update. Specifically, it constructs a dynamic graph with targets as nodes and spatiotemporal relationships as edges, models the relative positions, motion trajectories, and interaction relationships between targets, and identifies typical dynamic scene patterns.

[0050] Specifically, high-level behavioral semantics are extracted from the spatial-temporal relationship graph, and graph neural networks are used for modeling to identify complex interaction behavior patterns. The identification results of dynamic interaction relationships are output in probabilistic form and used for semantic reasoning of subsequent scene labels.

[0051] Specifically, complex interaction patterns include “avoidance” behavior between vehicles, “conflict” or “yield” relationships between pedestrians and vehicles, and group movement patterns among multiple targets (such as congestion formation and dissipation).

[0052] In one possible implementation, constructing a spatial-temporal relationship map includes:

[0053] The targets in the target detection results are used as nodes, atomic-level structured labels are used as node attributes, and spatial relative relationships, cross-frame temporal associations, and dynamic interaction modes between targets are used as edge relationships to construct a spatial-temporal relationship graph.

[0054] In one possible implementation, such as Figure 3 As shown, the construction process of the spatial-temporal relationship map includes:

[0055] Step 201: Use the Hungarian algorithm or a re-identification module based on appearance features to perform association matching on the same target in multiple consecutive frames of images, and establish cross-frame target correspondence;

[0056] Step 202: Use Kalman filtering or other recursive state estimation methods to smooth the associated target motion state and predict its trajectory to obtain continuous motion information of the target.

[0057] Step 203: Dynamically update the graph structure based on the association and prediction results, including adding and deleting nodes and adjusting edge relationships, to form a spatial-temporal relationship graph that can accurately reflect the dynamic changes of the scene.

[0058] In one possible implementation, using graph neural networks to model dynamic interactions in the graph includes: such as Figure 4 As shown, a graph convolutional network or graph attention network is used to aggregate the neighbor node information of each node through a multi-layer message passing mechanism, and learn the context representation of each target, i.e., the dynamic interaction relationship.

[0059] Step 103: Based on dynamic interaction relationships and traffic rule knowledge base, perform semantic reasoning on the spatial-temporal relationship graph, and output scene-level semantic labels as the final scene understanding result.

[0060] Specifically, the spatial-temporal relational graph is input to the semantic reasoning module, which includes a traffic rule knowledge base and a reasoning engine. First, scene matching is performed, matching the current graph with typical scene patterns in the knowledge base to identify potential scene types. Then, rule verification is performed, checking whether the current target behavior complies with traffic rules and identifying any violations or potential risks. Finally, scene-level semantic labels are generated, outputting high-level semantic descriptions with confidence scores, such as "intersection congestion, it is recommended to slow down" and "pedestrians crossing the street, yield to them," which are scene-level understanding results, i.e., scene labels. The generated scene labels not only contain semantic descriptions but also include confidence scores and time validity periods for use by the decision-making module. The reasoning process is highly interpretable, meeting the safety and transparency requirements of autonomous driving systems.

[0061] Specifically, the traffic rules knowledge base is organized using a graph-based or production-based rule structure, containing the following structured information: semantic interpretations of traffic signs and markings, including specific rules such as stop line observation requirements and the meaning of speed limit signs; right-of-way rules, covering basic traffic norms such as priority for main roads, pedestrian priority, and yielding to turning vehicles; and definitions of typical scenarios, including standard behavioral patterns for typical scenarios such as intersections, school zones, and construction zones. This knowledge base is constructed using an ontology-based approach, supports semantic retrieval and logical reasoning functions, and can adapt to differences in traffic regulations across different regions through an OTA (Over-The-Air) online update mechanism, ensuring continuous evolution capabilities.

[0062] In summary, in this embodiment of the invention, the zero-shot capability of the visual language large model is used to generate fine-grained atomic-level structured labels, and a dynamically evolving spatial-temporal graph is constructed based on these labels. Combined with a traffic rule knowledge base for reasoning, high-level scene labels that can be understood by humans are finally generated, which significantly improves the cognitive depth of the autonomous driving system for complex scenes.

[0063] In one possible implementation, the scene label generation method based on dynamic graph inference further includes a lighting condition analysis and confidence calibration step: inputting the environmental image into the lighting condition analysis module and outputting the lighting analysis result; and adaptively calibrating the confidence of the atomic-level structured label based on the lighting analysis result and the target detection result.

[0064] Specifically, based on a lighting condition analysis module and an attention mechanism, the lighting condition analysis module is used to identify and classify the lighting environment type of the current image in real time. This module operates in a lightweight manner, suitable for embedded deployment, and its output serves as an important basis for subsequent label credibility correction. The module performs multi-scale feature extraction on the input image and outputs the following analysis results:

[0065] Lighting type classification: The current frame image is divided into typical lighting categories such as "direct sunlight", "weak night light", "backlight", "interlaced shadows" and "standard daylight". The classification is achieved by a softmax classifier, which is trained on a large number of real driving images labeled with lighting conditions.

[0066] Image quality assessment metrics: A set of objective quality metrics are calculated simultaneously, including overall contrast, local brightness variance, edge sharpness, and signal-to-noise ratio, thereby quantifying the reliability of the current image signal.

[0067] Specifically, the confidence level adaptive calibration involves calibrating the original confidence level output by the target detection model based on the real-time illumination analysis results, including the following calibration methods:

[0068] Based on the preset basic confidence threshold, when it is determined that the target is under challenging lighting conditions such as low light or backlight, the confidence threshold is dynamically lowered through predefined rules or prediction network to avoid missing key targets. Under ideal lighting conditions, the threshold is appropriately raised to suppress false positive detections.

[0069] In poor lighting conditions, the tolerance of the nonmaximum suppression algorithm is increased, and the detection stability is improved by fusing detection results from multiple frames for voting decisions.

[0070] An illumination uncertainty score is attached to each generated scene label. This score is determined by the output of the illumination analysis module and the target detection confidence score, providing a reliability reference for the downstream decision-making module.

[0071] The closed-loop feedback correction and calibration method described above can maintain the consistency and reliability of tag generation under varying lighting conditions, significantly improving the overall robustness of the method.

[0072] In summary, in this embodiment of the invention, by using a dynamic correction mechanism for label confidence based on illumination adaptation, a dedicated CNN network is employed to analyze illumination conditions in real time, dynamically adjust the confidence threshold for target detection and the multi-frame fusion strategy, thereby improving the perception stability under different illumination conditions, effectively avoiding decision errors caused by light and shadow interference, and significantly improving robustness under different illumination environments.

[0073] In one possible implementation, the scene tag generation method based on dynamic graph inference further includes storing scene tags using a dual-channel architecture; storing scene tags using a dual-channel architecture includes:

[0074] The generated scene-level semantic tags are stored in a high-speed cache through the real-time channel, supporting low-latency access. The scene-level semantic tags and feature vectors are compressed and stored using feature distillation through the archive channel, supporting long-term data analysis and mining. A data synchronization manager is provided between the two channels, which is responsible for migrating the processed data in the real-time channel to the archive channel according to preset strategies (such as time-triggered or event-triggered), thus balancing the system's requirements for real-time performance and long-term data accumulation.

[0075] Specifically, the real-time channel is implemented based on high-speed memory or non-volatile memory and is used to store high-priority scene tags and their related metadata (such as timestamps, locations, and confidence levels) generated in the short term (e.g., the last 15 minutes). The data is stored in an uncompressed or lightly compressed format, supporting microsecond-level low-latency access, and is mainly used for real-time perception, behavior prediction, and instant decision-making of autonomous vehicles.

[0076] Specifically, the archive channel is designed for long-term data storage and large-scale historical data analysis, employing high-compression algorithms to compress scene tags and their corresponding feature vectors. To further save storage space, feature distillation technology is used to extract high-dimensional feature vectors into low-dimensional semantic feature codes, reducing the storage footprint by an order of magnitude while preserving semantic information to the maximum extent. Archived data is stored on low-cost, high-capacity media, supporting efficient indexing and retrieval by time, geographic location, or scene type.

[0077] In summary, in this embodiment of the invention, by designing a dual-track parallel storage architecture of "real-time channel + archive channel", the low-latency access requirements for real-time decision-making are guaranteed, and the efficient management of long-term data is achieved by utilizing feature distillation and high compression algorithms, which significantly reduces storage costs and accelerates the algorithm iteration process.

[0078] In another embodiment of the present invention, an electronic device is also provided. The electronic device includes a processor and a memory. The memory stores at least one instruction or at least one program. The at least one instruction or at least one program is loaded and executed by the processor to implement the scene label generation method based on dynamic graph reasoning proposed in the embodiments of the present invention.

[0079] The foregoing primarily describes the solutions provided by the embodiments of the present invention from the perspective of the device. It is understood that, in order to achieve the above functions, the device includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the algorithmic steps of the various examples described in the embodiments disclosed herein, the present invention can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0080] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions within the technical scope disclosed in the present invention should be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A scene tag generation method based on dynamic graph reasoning, characterized in that, include: Open set target detection is performed on the collected environmental images based on a large visual language model to obtain target detection results, and atomic-level structured labels are generated based on the target detection results; A spatial-temporal relationship graph is constructed based on the atomic-level structured tags of consecutive frames, and a graph neural network is used to dynamically interact and model the graph to generate dynamic interaction relationships. Based on the dynamic interaction relationships and traffic rule knowledge base, semantic reasoning is performed on the spatial-temporal relationship graph, and scene-level semantic labels are output as the final scene understanding results.

2. The scene label generation method based on dynamic graph reasoning according to claim 1, characterized in that, Before performing open-set target detection on the acquired environmental images based on the large visual language model, the following steps are also included: The acquired environmental images are preprocessed, and the preprocessing includes at least one of the following operations: image denoising, resolution normalization, lens distortion correction, and color space conversion.

3. The scene label generation method based on dynamic graph reasoning according to claim 1, characterized in that, The generation of atomic-level structured tags includes: Based on a multi-task learning framework, the detection results of the open set target detection are used to perform target detection, attribute prediction, and geometric estimation, and output atomic-level structured labels. The atomic-level structured labels include target category labels, spatial location information, physical size parameters, and semantic attribute descriptions.

4. The scene label generation method based on dynamic graph reasoning according to claim 1, characterized in that, The construction of the spatial-temporal relationship graph includes: Using the targets in the target detection results as nodes, the atomic-level structured labels as node attributes, and the spatial relative relationships, cross-frame temporal associations, and dynamic interaction modes between targets as edge relationships, a spatial-temporal relationship graph is constructed.

5. The scene label generation method based on dynamic graph reasoning according to claim 1, characterized in that, The process of constructing the spatial-temporal relationship graph includes: Perform association matching on the same target in multiple consecutive frames of images to establish cross-frame target correspondence; The motion state of the associated target is smoothed and its trajectory is predicted to obtain continuous motion information of the target. The graph structure is dynamically updated based on the association and prediction results, forming a spatial-temporal relationship graph that reflects the dynamic changes of the scene.

6. The scene label generation method based on dynamic graph reasoning according to claim 1, characterized in that, The step of using a graph neural network to perform dynamic interaction modeling of the graph includes: By employing graph convolutional networks or graph attention networks, and aggregating the neighbor node information of each node through a multi-layer message passing mechanism, the contextual representation of each target is learned.

7. The scene label generation method based on dynamic graph reasoning according to claim 1, characterized in that, The method further includes: The environmental image is input into the illumination condition analysis module, and the illumination analysis results are output. The confidence level of the atomically structured tag is adaptively calibrated based on the illumination analysis results and the target detection results.

8. The scene label generation method based on dynamic graph reasoning according to claim 1, characterized in that, The method also includes storing scene tags using a dual-channel architecture, including: The generated scene-level semantic tags are stored in a high-speed cache memory via a real-time channel; The scene-level semantic tags and feature vectors are compressed and stored using the feature distillation method through the archive channel.

9. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or at least one program being loaded and executed by the processor to implement the scene label generation method as described in any one of claims 1-8.

Images