Visual question and answer data set construction method for three-dimensional driving scene understanding

Abstract

The invention is suitable for the technical field of automatic driving, and provides a visual question and answer data set construction method for understanding a three-dimensional driving scene. The method mainly comprises the following five construction processes: perception data preprocessing, construction of a layered scene system and scene graph annotations, question set and answer normal form design, scene level question and answer pair generation and manual post-processing. According to the method, high-efficiency and high-quality automatic construction of scene-level question and answer data is realized, data set support is provided for multi-modal model training and landing special for an automatic driving three-dimensional scene understanding task, and new inspiration is brought to VQA and automatic driving communities.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and in particular relates to a method for constructing a visual question-answering dataset for understanding three-dimensional driving scenes. Background Art

[0002] Autonomous driving is a rapidly developing field with great potential to improve traffic safety and efficiency with the advancement of sensor technology and computer vision. However, the primary challenge in achieving autonomous driving is scene understanding, which involves navigating in complex, changing, and globally coordinated 3D scenes and directly affects the vehicle's future state prediction, driving behavior decision-making, and safe interaction with the environment. As traditional perception technologies such as 3D object detection and tracking continue to mature, autonomous driving systems are moving towards enhanced interpretability and flexible human-vehicle interaction. In this context, visual question answering (VQA) has attracted widespread attention due to its unique natural language question answering format and user-agent interaction performance. On the one hand, drivers and passengers can fully perceive and understand the surrounding environment and traffic through natural language, which enhances their trust in the system's perception capabilities. On the other hand, drivers and passengers can verify the correctness of the perception system in a timely manner through question and answer, which improves the user experience of interacting with the system.

[0003] There are various VQA datasets, including Image Question Answering (ImageQA) datasets and Video Question Answering (VideoQA) datasets. ImageQA focuses on understanding the content of two-dimensional static images. Early studies attempted to solve this task by fusing image features extracted by convolutional neural networks (CNNs) and encoding of questions. Recently, Transformer-based models have achieved state-of-the-art performance through large-scale visual-language pre-training. VideoQA focuses on mining contextual information from dynamic videos. Most studies use attention networks to extract question-guided spatiotemporal contextual information to improve the accuracy of question-answering data. Three-dimensional question answering (3D-QA) is a new task in the field of VQA, focusing on answering questions about three-dimensional scenes represented by multiple views. Unlike traditional two-dimensional question answering (2D-QA), 3D-QA requires understanding the geometric structure and spatial relationship of objects in three-dimensional scenes, which places higher-level requirements on the model.

[0004] Although the VQA community has made great progress, models trained on existing VQA datasets still have limitations in coping with the complexity of autonomous driving scenarios. This limitation mainly stems from the differences in visual data and question-answer pairs between autonomous driving scenarios and existing VQA benchmarks. This requires actively exploring the expansion of visual data from a single form of global view of the road or local view of the vehicle to a global and local collaborative form, expanding question-answer pairs from two-dimensional static situational perception to comprehensive three-dimensional dynamic spatiotemporal scene reasoning, and requiring models to effectively utilize the complementary information of multiple types of perception data to understand complex scenes and capture object dynamics. In addition, most VQA datasets only provide closed-ended question-answer pair annotations with simple Boolean answers (i.e., yes or no) or limited multiple-choice responses (e.g., stop, move, and park), which are often insufficient in scale and richness, and require a lot of manpower and time costs. Therefore, the scarcity of VQA datasets directly hinders the exploration of various challenging visual-language tasks in the field of autonomous driving. Summary of the invention

[0005] The purpose of an embodiment of the present invention is to provide a method for constructing a visual question answering dataset for three-dimensional driving scene understanding, aiming to solve the problems raised in the above background technology.

[0006] The embodiment of the present invention is implemented as follows: a method for constructing a visual question answering dataset for three-dimensional driving scene understanding, comprising the following steps: Step 1: Perception data preprocessing; Automated image stitching technology is used to merge multi-view scene images into full-view form and eliminate obvious overlapping areas; the overall workflow of image stitching is as follows: Firstly, SIFT algorithm is used to detect and match feature points. Secondly, the transformation parameters are calculated using the matched feature points, and the common image plane is defined by combining the transformation parameters and image size. Finally, images are merged based on the stitching strategy.

[0007] Step 2: Construction of hierarchical scene system and scene graph annotation; A standard hierarchical scenario system and scenario graph annotations are constructed to represent driving scenarios. The hierarchical scenario system covers four scenario elements: weather conditions, road conditions, traffic facilities, and traffic participants, and defines their attributes. The scenario graph annotations are used as a concrete representation of the hierarchical scenario system. The central node represents the vehicle, the middle layer nodes represent the four types of scenario elements, and the outermost nodes represent the various attributes of the scenario elements. The vehicle, scenario elements, and attribute nodes are connected by relationship edges representing actions (verbs) or spatial relationships (prepositions) to standardize them.

[0008] Step 3: Question set and answer paradigm design; Based on the hierarchical scenario system, four sets of question sets were manually designed, which correspond to the four scene elements respectively and contain diverse synonymous expressions; based on the scene graph annotations, four sets of answer paradigms were also manually designed, which match the four sets of question sets respectively and contain various attribute parameterizations.

[0009] Step 4: Generate scenario-level question-answer pairs; We use a scalable GPT-4V fully automatic programmatic question-answer generation pipeline that takes full-view scene images and BEV map images as visual inputs and combines question sets and answer paradigms as contextual information to prompt GPT-4V to automatically generate scene-level question-answer pairs.

[0010] Step 5: Manual post-processing; In order to effectively solve the problem of hallucination tendency of GPT-4V, the annotations of the generated multi-round question and answer pairs were manually post-processed, including deleting inappropriate questions and answers, modifying incorrect expressions, and supplementing missing options. ChatGPT was further used to check and correct common grammatical errors.

[0011] A further technical solution is that in step 1, a large number of multi-view scene images are extracted from the nuScenes original data. , is the number of images, in order to effectively merge images , and eliminate the overlapping parts , using an automated image stitching technology, the overall workflow of image stitching is as follows: First, the SIFT algorithm is used to detect and match feature points. The algorithm includes a feature detector and a feature descriptor. The feature detector is used to detect multi-view scene images. The line, contour and edge feature points in , and the feature descriptors are used to describe the properties of these points: (1) in, represents a set of feature points, represents the corresponding feature descriptor set, is the number of feature points.

[0012] Feature point matching is based on the similarity between feature descriptors. In the matching process, in order to ensure the uniqueness of the matching relationship, the ratio test method is used to screen feature points, and only features with sufficient differentiation from the nearest neighbor features are retained, that is, the ratio of the matching distance to the next nearest neighbor matching distance is less than the threshold : (2) in, Represents the set of matched feature points, and Represents two matching feature points, represents the Hamming distance between feature descriptors, and Respectively The nearest and second nearest feature identifiers; Then, the transformation parameters are calculated using the matched feature points. Two-dimensional similarity transformation is used for projection to achieve image movement, scaling, and rotation without changing the image content. RANSAC regression is used to estimate the transformation matrix and identify and remove outliers: (3) in, represents the homography matrix, Represents the transformation matrix. The combination of transformation parameters and image size can define a common image plane: (4) in, represents a regular rectangular plane, onto which the input image will be transformed. Represents a rectangular clipping operation; Finally, the images are merged based on the stitching strategy. In the overlapping area, the stitching line that cuts the least image content is found, and the images are "stitched" along this line. The pixel values ​​on both sides of the stitching line are transferred from the adjacent single images to the merged image to ensure that it is consistent with the input image in geometry and photometry: (5) (6) in, represents the optimal suture line, Represents the merged full-view scene image.

[0013] According to a further technical solution, step 2 comprises the following specific steps: Step 2.1: Build a hierarchical scenario system ; Layered scene system Used to describe real driving scenarios, which cover four scenario elements: ;Specifically: Weather conditions : As an important element in driving scenarios, weather conditions reflect driving difficulty; define three attributes of weather: , and ; Including sunny, cloudy, rainy, foggy and snowy days; Includes day and night; The attributes are classified as low, medium and high to quantify the potential driving risk; and there is a correlation between these attributes; Road conditions : As a basic element in the driving scene, road conditions determine the driving style; four attributes of the road are defined: , , and ; Includes urban roads, suburban roads, rural roads and highways; Covering single lane, dual lane, multi-lane, intersections and unstructured roads with specific traffic flow patterns; Distinguish between one-way and two-way, which is related to lane selection and safe lane change; It can be summarized as straight, left turn, right turn, straight + left turn, straight + right turn and all directions to determine the expected driving mode of the vehicle; Transportation Facilities : As a key element in the driving scene, traffic facilities regulate driving behavior; focus on three types of traffic facilities in the driving process: traffic signs, road markings and traffic lights, and define their inherent properties; for traffic signs, consider their specific types ; For road markings, analyze their directions ; For traffic lights, describe their color status , including red, yellow and green; Traffic Participants :As the core element in the driving scene, traffic participants affect the driving state; referring to the cognitive mode of human drivers during driving, only the key traffic participants within a specific detection range are focused on, and their five attributes are defined: , , , and ; That is, the specific category of traffic participants; That is, the status of traffic participants; The 3D bounding boxes of traffic participants are recorded and labeled ; Involving traffic participants at the current observation time and the past The coordinates of the time steps are marked as , ; Six types of traffic participants’ orientations relative to the vehicle are defined, namely front, rear, front left, front right, rear left, and rear right. Next, the angle between the traffic participant and the vehicle’s forward direction is calculated. : (7) in, Represents the 3D bounding box of the ego vehicle; based on the angle range, the orientation relationship is defined as follows: (8) The vehicle's forward direction is defined as , and the clockwise angle is positive; Based on the above five attributes, two types of characteristics are constructed for each traffic participant: 3D inherent characteristics and 2D motion characteristics ; 3D inherent characteristics Describes the static properties of traffic participants, including their category, 3D size, status, position coordinates, and orientation relationship in the current frame; 2D motion characteristics It describes the dynamic behavior of traffic participants in the historical time range, involving the following two cases: a) if the traffic participant remains stationary, only its position coordinates in the current frame are considered; b) if the traffic participant moves, its position coordinates in the historical frame and the current frame are considered; Step 2.2: Scene graph annotation; Construct scene graph annotations to visualize the proposed hierarchical scene system: the central node represents the ego vehicle, the middle layer nodes represent the four types of scene elements, and the outermost nodes represent the various attributes of the scene elements; the ego vehicle, scene elements, and attribute nodes are connected by relationship edges representing actions or spatial relationships; The scene graph is annotated with natural language and identifiers. In order to utilize the scene graph annotations to generate answer paradigms, it is normalized. Specifically, first, based on the concept of hierarchical scene system, an ontology list is integrated and created, covering scene elements, attributes, and relationships. Then, based on category rules, the construction of the scene graph is manually planned, and the graph content is further enriched using semantic attributes, object detection confidence, co-occurrence statistics, and word embedding techniques. Finally, inaccurate or unnatural edges are pruned to obtain a scene graph with unified and clear nodes and relationship edges. A further technical solution is that in step 3, a set of questions is manually designed based on a hierarchical scenario system. , including the following four groups , which correspond to four scene elements respectively; in order to prevent the model from overfitting fixed question patterns, each set of questions should contain multiple synonymous expressions of a question option, and at least two different expression sentences should be created for each question pattern; finally, four sets of grammatically correct, diverse and highly relevant question sets were obtained; Designing answer paradigm based on scene graph annotations : First, traverse each node and relationship edge in the scene annotation graph and combine them into a triple form of [element-relationship-attribute]; then, write a fixed text for each triple; finally, generate four sets of answer paradigms: , which are matched to four sets of questions and contain various attribute parameterizations.

[0014] A further technical solution is that in step 4, a scalable GPT-4V fully automatic programmatic pipeline is used to generate multiple rounds of question-answer pairs; the pipeline converts the full-view scene image and BEV map images As visual input, combined with a set of questions and answer paradigm As context information, prompt GPT-4V to automatically generate scene-level question-answer pairs ; Specifically, GPT-4V randomly selects a question option from each set of questions through the optimal sampling strategy, infers the true answer on the full-view scene image and BEV map image, and outputs an instantiated answer paradigm; when generating question-answer pairs around traffic participants, the acquired nuScenes original annotations and manually defined orientation relationships are also provided to GPT-4V as privileged information to assist its reasoning.

[0015] The embodiment of the present invention provides a method for constructing a visual question answering dataset for three-dimensional driving scene understanding, and its beneficial effects are as follows: (1) An automated image stitching technique is used to merge multi-view scene images into a full-view form and eliminate obvious overlapping areas. The technique uses a feature-based approach to model the relationship between the partial images and align them in a common image plane. During the merging process, a stitching strategy is used to ensure that the elements in the image remain as intact as possible.

[0016] (2) A standard hierarchical scenario system and scenario graph annotation are constructed to achieve hierarchical and graph-structured representation of driving scenarios. The hierarchical scenario system covers four scenario elements: weather conditions, road conditions, traffic facilities, and traffic participants. The scenario graph is used as a concrete representation of the hierarchical scenario system. The central node represents the vehicle, the middle layer nodes represent the four types of scenario elements, and the outermost nodes represent the various attributes of the scenario elements. The vehicle, scenario elements, and attribute nodes are connected by relationship edges representing actions (verbs) or spatial relationships (prepositions).

[0017] (3) A scalable GPT-4V fully automatic programmatic question-answer generation pipeline is proposed. This pipeline takes full-view scene images and BEV map images as visual input, and combines question sets and answer paradigms as contextual information to prompt GPT-4V to automatically generate scene-level question-answer pairs. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of a method for constructing a visual question answering dataset for three-dimensional driving scene understanding provided by an embodiment of the present invention; Figure 2 The overall workflow for image stitching; Figure 3 Visualize the scene graph; Figure 4 for Figure 4 Qualitative demonstration of the nuScene3D dataset. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0020] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0021] like Figure 1 As shown, a method for constructing a visual question answering dataset for three-dimensional driving scene understanding provided by an embodiment of the present invention includes the following steps: Step 1: Perception data preprocessing; The automatic image stitching technology is used to merge the multi-view scene images into a full-view form and eliminate obvious overlapping areas. The merged full-view scene image can effectively avoid the model's neglect and misunderstanding of scene details. The working principle of image stitching technology is as follows: The relationship between the parts of the image is modeled through a feature-based approach and aligned in a common image plane. During the merging process, a stitching strategy is used to ensure that the elements in the image remain as intact as possible. The overall workflow of image stitching consists of three steps: Firstly, SIFT algorithm is used to detect and match feature points. Secondly, the transformation parameters are calculated using the matched feature points, and the common image plane is defined by combining the transformation parameters and image size. Finally, images are merged based on the stitching strategy.

[0022] Step 2: Construction of hierarchical scene system and scene graph annotation; Construct a standard hierarchical scenario system and scene graph annotations to represent driving scenarios in a hierarchical and graph-structured manner. The hierarchical scenario system covers four scene elements: weather conditions, road conditions, traffic facilities, and traffic participants, and defines their various attributes. The scene graph annotations serve as a concrete representation of the hierarchical scenario system. The central node represents the vehicle, the middle layer nodes represent the four types of scene elements, and the outermost nodes represent the various attributes of the scene elements. The vehicle, scene elements, and attribute nodes are connected by relationship edges representing actions (verbs) or spatial relationships (prepositions), and are standardized.

[0023] Step 3: Question set and answer paradigm design; Based on the hierarchical scenario system, four sets of question sets were manually designed, which correspond to the four scenario elements and contain diverse synonymous expressions to prevent the model from overfitting fixed question patterns. Based on the scene graph annotations, four sets of answer paradigms were also manually designed, which match the four sets of question sets and contain various attribute parameterizations.

[0024] Step 4: Generate scenario-level question-answer pairs; We use a scalable GPT-4V fully automatic programmatic question-answer generation pipeline that takes full-view scene images and BEV map images as visual inputs and combines question sets and answer paradigms as contextual information to prompt GPT-4V to automatically generate scene-level question-answer pairs.

[0025] Step 5: Manual post-processing; In order to effectively solve the problem of hallucination tendency of GPT-4V, the annotations of the generated multi-round question and answer pairs were manually post-processed, including deleting inappropriate questions and answers, modifying incorrect expressions, and supplementing missing options. ChatGPT was further used to check and correct common grammatical errors.

[0026] As a preferred embodiment of the present invention, in step 1, the multi-view scene image is converted into the expected full-view form to avoid neglecting and misunderstanding the details of the scene image. A large number of multi-view scene images are extracted from the nuScenes original data. , is the number of images, which are collected by on-board cameras (1600×1200 pixels, 12FPS) with different viewing angles. However, there are obvious overlapping areas between them. In order to effectively merge images , and eliminate the overlapping parts , an automated image stitching technique is used: the relationship between the image parts is modeled through a feature-based approach and aligned in a common image plane. During the merging process, a stitching strategy is used to ensure that the elements in the image remain as intact as possible. Figure 2 The overall workflow of image stitching is demonstrated.

[0027] First, the SIFT algorithm is used to detect and match feature points. The algorithm includes a feature detector and a feature descriptor. The feature detector is used to detect multi-view scene images. The line, contour and edge feature points in , and the feature descriptors are used to describe the properties of these points: (1) in, represents a set of feature points, represents the corresponding feature descriptor set, is the number of feature points. Feature point matching is based on the similarity between feature descriptors. In the matching process, in order to ensure the uniqueness of the matching relationship, the ratio test method is used to screen feature points, and only features with sufficient differentiation from the nearest neighbor features are retained, that is, the ratio of the matching distance to the next nearest neighbor matching distance is less than the threshold : (2) in, Represents the set of matched feature points, and Represents two matching feature points, represents the Hamming distance between feature descriptors, and Respectively The nearest and second nearest feature identifiers; Then, the transformation parameters are calculated using the matched feature points. The projection used is a two-dimensional similarity transformation that can move, scale, and rotate the image without changing the image content. However, the number of feature points often exceeds the degrees of freedom required for the transformation, and incorrectly matched feature points may affect the accuracy of the parameters. Therefore, a robust RANSAC regression is used to estimate the transformation matrix and effectively identify and remove outliers: (3) in, represents the homography matrix, Represents the transformation matrix. The combination of transformation parameters and image size can define a common image plane: (4) in, represents a regular rectangular plane, onto which the input image will be transformed. Represents a rectangular clipping operation; Finally, the images are merged based on the stitching strategy. In the overlapping area, a stitching line is found that tries not to cut the image content, and the images are "stitched" along this line. The pixel values ​​on both sides of the stitching line are transferred from the adjacent single images to the merged image to ensure that it is consistent with the input image in geometry and photometry: (5) (6) in, represents the optimal suture line, Represents the merged full-view scene image.

[0028] As a preferred embodiment of the present invention, step 2 includes the following specific steps: Step 2.1: Build a hierarchical scenario system ; Layered scene system Used to describe real driving scenarios, which cover four scenario elements: . Specifically: Weather conditions : As an important element in driving scenarios, weather conditions reflect driving difficulty. Three attributes of weather are defined: , and . Covers sunny, cloudy, rainy, foggy and snowy days; Includes day and night; The attributes are classified as low, medium, and high to quantify the potential driving risk. It is worth noting that these attributes are interrelated. For example, when driving in rainy / snowy weather or at night, a more cautious driving strategy is required due to reduced visibility.

[0029] Road conditions : As a basic element in the driving scene, road conditions determine the driving style. Four attributes of the road are defined: , , and . It contains urban roads, suburban roads, rural roads and highways, presenting unique driving challenges; Covering single lane, dual lane, multi-lane, intersections and unstructured roads with specific traffic flow patterns; It is divided into one-way and two-way, which is related to lane selection and safe lane changing. It can be summarized as straight, left turn, right turn, straight + left turn, straight + right turn and all directions to determine the expected driving mode of the vehicle.

[0030] Transportation facilities : As a key element in the driving scene, traffic facilities regulate driving behavior. We focus on three common traffic facilities in the driving process: traffic signs, road markings and traffic lights, and define their inherent properties. For traffic signs, consider their specific types , such as stop signs and speed limit signs; for road markings, analyze their directions , such as straight-ahead markings, sidewalk markings, etc.; for traffic lights, describe their color status , including red, yellow and green. Each traffic facility has the function of providing normative guidance for driving behavior.

[0031] Traffic Participants :As the core element in the driving scene, traffic participants affect the driving state. Referring to the cognitive mode of human drivers during driving, we only focus on key traffic participants within a specific detection range and define their five attributes: , , , and . Covers cars, adults and animals, etc.; This includes parking, moving and standing; The 3D bounding boxes of traffic participants (labeled as ); Involving traffic participants at the current observation time and the past The coordinates of the time step (denoted by , ). These attributes can be directly obtained from the original nuScenes annotations. However, the original annotations do not provide the orientation relationship of traffic participants relative to the ego vehicle. Given that relative spatial relationships are crucial in driving scenarios, a method to determine The following rules define six orientation relationships: front, rear, front left, front right, rear left, and rear right. Next, the angle between the traffic participant and the vehicle's forward direction is calculated. : (7) in, Represents the 3D bounding box of the ego vehicle. Based on the angle range, the orientation relationship is defined as follows: (8) The vehicle's forward direction is defined as , and clockwise angles are positive.

[0032] Based on the above five attributes, two types of characteristics are constructed for each traffic participant: 3D inherent characteristics and 2D motion characteristics 3D Intrinsic Properties Describes the static properties of traffic participants, including their category, 3D size, status, position coordinates, and orientation relationship in the current frame. It describes the dynamic behavior of traffic participants within the historical time range, involving the following two situations: a) if the traffic participant remains stationary, only its position coordinates in the current frame are considered; b) if the traffic participant moves, its position coordinates in the historical frame and the current frame are considered.

[0033] Step 2.2: Scene graph annotation; The scene graph annotation is constructed to visualize the proposed hierarchical scene system: the central node represents the ego vehicle, the middle layer nodes represent the four types of scene elements, and the outermost nodes represent the various attributes of the scene elements. The ego vehicle, scene elements, and attribute nodes are connected by relationship edges representing actions (verbs) or spatial relationships (prepositions).

[0034] The scene graph is annotated with natural language and identifiers. In order to utilize the scene graph annotations to generate answer paradigms, they are normalized. First, based on the concept of hierarchical scene system, a clear and explicit list of ontologies covering scene elements, attributes, and relationships is integrated and created. Then, based on category rules, the construction of the scene graph is manually curated, and the graph content is further enriched using semantic attributes, object detection confidence, co-occurrence statistics, and word embedding techniques. Finally, inaccurate or unnatural edges are pruned to obtain a scene graph with clear, unified, rich, and explicit node and relationship edges. Figure 3 Provides a visual representation of scene graph annotations.

[0035] As a preferred embodiment of the present invention, in step 3, a set of questions is manually designed based on a hierarchical scenario system. , including the following four groups , which correspond to the four scene elements respectively. In order to prevent the model from overfitting fixed question patterns, each set of questions should contain multiple synonymous expressions of a question option. To this end, various grammatical rules were queried and a variety of synonymous question patterns were collected from them, including interrogative sentences starting with various words and declarative sentences expressing requests. At least two different expression sentences were created for each question pattern. Finally, four sets of grammatically correct, diverse and highly relevant question sets were obtained.

[0036] Designing answer paradigm based on scene graph annotations Relatively simple: First, traverse each node and relationship edge in the scene annotation graph and combine them into a triple of [element-relationship-attribute]. Then, write a fixed text for each triple. Finally, four sets of answer paradigms are generated: , which are matched to four sets of questions and contain various attribute parameterizations.

[0037] As a preferred embodiment of the present invention, in step 4, a scalable GPT-4V fully automatic programmatic pipeline is used to generate multiple rounds of question-answer pairs. The pipeline converts the full-view scene image and BEV map images As visual input, combined with a set of questions and answer paradigm As context information, prompt GPT-4V to automatically generate scene-level question-answer pairs Specifically, GPT-4V randomly selects a question option from each set of questions through the optimal sampling strategy, infers the true answer on the full-view scene image and BEV map image, and outputs an instantiated answer paradigm. It is worth noting that when generating question-answer pairs around traffic participants, this method also provides the obtained nuScenes original annotations and manually defined orientation relationships as privileged information to GPT-4V to assist its reasoning.

[0038] As a preferred embodiment of the present invention, in step 5, a known challenge of GPT-4V is the tendency to hallucinate, which is manifested as an inaccurate match between the generated text and the visual content. For example, when processing an intersection scene, GPT-4V may mistakenly identify a pedestrian signal as a traffic signal, which may cause the vehicle to make a completely unreasonable decision-making behavior. Manual post-processing effectively solves this problem. This method uses 5 annotators to strictly correct the generated question and answer pairs, including deleting inappropriate questions and answers, modifying incorrect expressions, and supplementing missing options. In addition, ChatGPT is further used to check and correct common grammatical errors.

[0039] As a preferred embodiment of the present invention, the constructed visual question answering dataset is named nuScene3D. In order to demonstrate its unique advantages, a large number of statistical analyses are performed on the nuScene3D dataset; in order to verify the effectiveness and robustness of this method in the task of autonomous driving 3D scene understanding, a general visual language model (VLM) is used to conduct test evaluation on the nuScene3D dataset; finally, a visualization sample of the nuScene3D dataset is provided.

[0040] (1) Dataset comparison A comprehensive statistical analysis of the nuScene3D dataset is performed, and its comparison with other VQA datasets is summarized in Table 1. The nuScene3D dataset constructed by this method has the following advantages: 1) This dataset is a large-scale VQA dataset, containing 34,000 diverse visual scenes and 240K question-answer pairs, with an average of 7 question-answer pairs per scene; 2) In terms of visual type, this dataset combines multi-view scene images and BEV map images (multi-frame), which requires mining temporal and spatial information, increasing the complexity of the benchmark; 3) Question-answer pairs are collected automatically, saving a lot of manpower and time costs; 4) The data focuses on outdoor driving scenes, providing higher challenges for perception and reasoning tasks.

[0041] Table 1 Comparison of nuScene3D with other VQA datasets

[0042] In summary, the nuScene3D dataset has advantages in terms of visual type, data scale, and content, demonstrating the unique advantages of the present invention and helping to promote the research and development of VQA in the field of autonomous driving.

[0043] (2) Evaluation of 3D scene understanding tasks Existing work lacks a VLM-based baseline method for the 3D scene understanding task. In order to verify the effectiveness and robustness of this method in this task, a general VLM was selected for evaluation. Video-LLaVA is a novel VLM that can jointly encode dynamic video and static images, which is highly compatible with the format of the nuScene3D dataset. It is used as a baseline model and fine-tuned using the nuScene3D dataset. The BLEU4, ROUGE, METEOR, and CIDEr metrics commonly used in the NLP community are used to evaluate the quality of text generation, where BLEU4 focuses on the accuracy of n-grams, ROUGE and METEOR consider the subtle differences in semantics and syntax, and CIDEr emphasizes the consensus and relevance of image descriptions. Next, the fine-tuning implementation details will be introduced in detail.

[0044] Visual data processing: Each BEV map image is resized to . We also uniformly sample 8 frames from a large number of full-view scene images (after image stitching) and process them into video form. The visual data in each batch is a combination of images and videos.

[0045] Video-LLaVA fine-tuning: The AdamW optimizer with cosine learning rate scheduling is used, the initial learning rate is set to 2e-5, the warm-up ratio is 0.03, and the gradient accumulation step is 2. The model is trained for 3 epochs, and the whole process is completed on 8 L20 (48GB) GPUs, with a batch size of 4 per GPU.

[0046] Table 2 provides the quantitative results of Video-LLaVA on the nuScene3D dataset. Video-LLaVA achieved scores of 58.9, 61.5, 78.4, and 288.9 on the four indicators of METEOR, BLEU4, ROUGE, and CIDEr, respectively, demonstrating excellent performance. The nuScene3D dataset and evaluation results can provide a broad space for exploration in subsequent research on 3D scene understanding for autonomous driving.

[0047] Table 2 Quantitative results of Video-LLaVA on the nuScene3D dataset

[0048] (3) Qualitative demonstration In order to gain a deeper understanding of the nuScene3D dataset, a visualization of the dataset samples was performed, such as Figure 4 As shown, the figure shows the BEV map, multi-view scenes and multi-round question-answer pairs, which fully demonstrates the challenge of the nuScene3D dataset in the task of three-dimensional scene understanding of autonomous driving, and further demonstrates the rationality of the present invention.

[0049] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing a visual question answering dataset for three-dimensional driving scene understanding, characterized in that: The following steps are involved: Step 1: Perception data preprocessing; Automated image stitching technology is used to merge multi-view scene images into full-view form and eliminate obvious overlapping areas; the overall workflow of image stitching is as follows: Firstly, SIFT algorithm is used to detect and match feature points. Secondly, the transformation parameters are calculated using the matched feature points, and the common image plane is defined by combining the transformation parameters and image size. Finally, the images are merged based on the stitching strategy. Step 2: Construction of hierarchical scene system and scene graph annotation; A standard hierarchical scenario system and scene graph annotations are constructed to represent driving scenarios. The hierarchical scenario system covers four scene elements: weather conditions, road conditions, traffic facilities, and traffic participants, and defines their attributes. The scene graph annotations are used as a concrete representation of the hierarchical scenario system. The central node represents the vehicle, the middle layer nodes represent the four types of scene elements, and the outermost nodes represent the various attributes of the scene elements. The vehicle, scene elements, and attribute nodes are connected by relationship edges representing actions or spatial relationships, and are standardized. Step 3: Question set and answer paradigm design; Based on the hierarchical scenario system, four sets of questions were manually designed, corresponding to the four scenario elements and containing a variety of synonymous expressions. Based on the scene graph annotations, four sets of answer paradigms were manually designed, matching the four sets of questions and containing various attribute parameterizations. Step 4: Generate scenario-level question-answer pairs; Use the scalable GPT-4V fully automatic programmatic question-answer generation pipeline, which takes full-view scene images and BEV map images as visual input, and combines question sets and answer paradigms as contextual information to prompt GPT-4V to automatically generate scene-level question-answer pairs; Step 5: Manual post-processing; In order to effectively solve the problem of hallucination tendency of GPT-4V, the annotations of the generated multi-round question and answer pairs were manually post-processed, including deleting inappropriate questions and answers, modifying incorrect expressions, and supplementing missing options. ChatGPT was further used to check and correct common grammatical errors.

2. The method for constructing a visual question answering dataset for three-dimensional driving scene understanding according to claim 1, characterized in that: In step 1, a large number of multi-view scene images are extracted from the nuScenes raw data. , For the number of images, an automated image stitching technique is used to merge the images , and eliminate the overlapping parts ,The overall workflow of image stitching is as follows: First, the SIFT algorithm is used to detect and match feature points. The algorithm includes a feature detector and a feature descriptor. The feature detector is used to detect multi-view scene images. The line, contour and edge feature points in , and the feature descriptors are used to describe the properties of these points: （1）； in, represents a set of feature points, represents the corresponding feature descriptor set, is the number of feature points; Feature point matching is based on the similarity between feature descriptors. In the matching process, in order to ensure the uniqueness of the matching relationship, the ratio test method is used to screen feature points, and only features with sufficient differentiation from the nearest neighbor features are retained, that is, the ratio of the matching distance to the next nearest neighbor matching distance is less than the threshold. : （2）； in, Represents the set of matched feature points, and Represents two matching feature points, represents the Hamming distance between feature descriptors, and Respectively The nearest and second nearest feature identifiers; Then, the transformation parameters are calculated using the matched feature points. Two-dimensional similarity transformation is used for projection to achieve image movement, scaling, and rotation without changing the image content. RANSAC regression is used to estimate the transformation matrix and identify and remove outliers: （3）； in, represents the homography matrix, Represents a transformation matrix; by combining the transformation parameters and the image size, a common image plane can be defined: （4）； in, represents a regular rectangular plane, onto which the input image will be transformed. Represents a rectangular clipping operation; Finally, the images are merged based on the stitching strategy; in the overlapping area, the stitching line that cuts the least on the image content is found, and the images are "stitched" along this line; the pixel values ​​on both sides of the stitching line are transferred from the adjacent single images to the merged image to ensure that it is consistent with the input image in geometry and photometry: （5）； （6）； in, represents the optimal suture line, Represents the merged full-view scene image.

3. The method for constructing a visual question answering dataset for three-dimensional driving scene understanding according to claim 2, characterized in that: The step 2 comprises the following specific steps: Step 2.1: Construct a hierarchical scenario system S; Layered scene system Used to describe real driving scenarios, which cover four scenario elements: ;Specifically: Weather conditions : Define three attributes of weather: , and ; Including sunny, cloudy, rainy, foggy and snowy days; Includes day and night; The attributes are classified as low, medium and high to quantify the potential driving risk; and there is a correlation between these attributes; Road conditions : Define four attributes of the road: , , and ; Includes urban roads, suburban roads, rural roads and highways; Covering single lane, dual lane, multi-lane, intersections and unstructured roads with specific traffic flow patterns; Differentiate between one-way and two-way, which is related to lane selection and safe lane change; It can be summarized as straight, left turn, right turn, straight + left turn, straight + right turn and all directions to determine the expected driving mode of the vehicle; Transportation facilities : Focus on three types of traffic facilities in the driving process: traffic signs, road markings and traffic lights, and define their inherent properties; for traffic signs, consider their specific types, namely ; For road markings, analyze their directions, i.e. ; For traffic lights, describe their color status, that is ; Traffic Participants : Refer to the cognitive mode of human drivers during driving, focus only on key traffic participants within a specific detection range, and define their five attributes: , , , and ; That is, the specific category of traffic participants; That is, the status of traffic participants; The 3D bounding boxes of traffic participants are recorded and labeled ; Involving traffic participants at the current observation time and the past The coordinates of the time steps are marked as , ; Define the six orientations of traffic participants relative to the vehicle, namely front, back, front left, front right, back left, and back right; next, calculate the angle between the traffic participant and the vehicle's forward direction : （7）； in, Represents the 3D bounding box of the ego vehicle; based on the angle range, the orientation relationship is defined as follows: （8）； The vehicle's forward direction is defined as , and the clockwise angle is positive; Based on the above five attributes, two types of characteristics are constructed for each traffic participant: 3D inherent characteristics and 2D motion characteristics ; 3D inherent characteristics Describes the static properties of traffic participants, including their category, 3D size, status, position coordinates, and orientation relationship in the current frame; 2D motion characteristics It describes the dynamic behavior of traffic participants in the historical time range, involving the following two cases: a) if the traffic participant remains stationary, only its position coordinates in the current frame are considered; b) if the traffic participant moves, its position coordinates in the historical frame and the current frame are considered; Step 2.2: Scene graph annotation; Construct scene graph annotations to visualize the proposed hierarchical scene system: the central node represents the ego vehicle, the middle layer nodes represent the four types of scene elements, and the outermost nodes represent the various attributes of the scene elements; the ego vehicle, scene elements, and attribute nodes are connected by relationship edges representing actions or spatial relationships; The scene graph is annotated with natural language and identifiers. In order to generate the answer paradigm using the scene graph annotations, it is normalized. Specifically: first, based on the concept of hierarchical scene system, an ontology list is integrated and created, covering scene elements, attributes and relationships. Then, based on category rules, the construction of the scene graph is manually planned, and the graph content is further enriched using semantic attributes, object detection confidence, co-occurrence statistics and word embedding techniques. Finally, inaccurate or unnatural edges are pruned to obtain a scene graph with unified and clear nodes and relationship edges.

4. The method for constructing a visual question answering dataset for three-dimensional driving scene understanding according to claim 3, characterized in that: In step 3, a set of questions is manually designed based on the hierarchical scenario system. , including the following four groups , which correspond to four scene elements respectively; in order to prevent the model from overfitting fixed question patterns, each set of questions contains multiple synonymous expressions of a question option, and at least two different expression sentences are created for each question pattern; finally, four sets of grammatically correct, diverse and highly relevant question sets are obtained; Designing answer paradigm based on scene graph annotations : First, traverse each node and relationship edge in the scene annotation graph and combine them into a triple form of [element-relationship-attribute]; then, write a fixed text for each triple; finally, generate four sets of answer paradigms: , which are matched to four sets of questions and contain various attribute parameterizations.

5. The method for constructing a visual question answering dataset for three-dimensional driving scene understanding according to claim 4, characterized in that: In step 4, a scalable GPT-4V fully automatic programmatic pipeline is used to generate multiple rounds of question-answer pairs; the pipeline converts the full-view scene image and BEV map images As visual input, combined with a set of questions and answer paradigm As context information, prompt GPT-4V to automatically generate scene-level question-answer pairs ; Specifically, GPT-4V randomly selects a question option from each set of questions through the optimal sampling strategy, infers the true answer on the full-view scene image and BEV map image, and outputs an instantiated answer paradigm; when generating question-answer pairs around traffic participants, the acquired nuScenes original annotations and manually defined orientation relationships are also provided to GPT-4V as privileged information to assist its reasoning.

Images