A method and system for generating virtual camera perspectives for autonomous driving simulation

Abstract

This invention discloses a method and system for generating virtual camera perspectives for autonomous driving simulation, belonging to the field of autonomous driving simulation technology. The method first acquires the vehicle's operating state, scene state, and initial observation images in the simulation environment, organizes them into an initial observation image set, and inputs it into a visual language model. The model then performs joint scene understanding, determining whether the image information meets the task requirements; if not, it generates structured virtual perspective requirements. Pose anchor points are established based on the vehicle's or onboard camera's pose, and relative perspective parameters are obtained by matching perspective parameter templates. The virtual camera target pose is calculated, and after spatial constraint checks, supplementary images are acquired and fed back to the model, forming a closed-loop iteration until the image information meets the task requirements or the maximum number of iterations is reached. Finally, the scene understanding result is output. This invention can automatically generate efficient supplementary perspectives, improving the completeness and accuracy of autonomous driving simulation scene understanding and adapting to complex simulation perception task requirements.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving simulation technology, and in particular to a method and system for generating virtual camera perspectives for autonomous driving simulation. Background Technology

[0002] With the development of autonomous driving technology and intelligent transportation systems, higher demands are being placed on vehicles' spatial understanding, information acquisition, and risk area observation capabilities in complex traffic environments. In autonomous driving simulation testing, scene understanding verification, and driver assistance decision analysis, image information serves as a crucial data foundation for systems to identify road structure, traffic participants, obstacle distribution, and potential conflict relationships. In recent years, the application of visual language models in autonomous driving scene understanding has been increasing, enabling systems to perform semantic descriptions, spatial relationship analysis, and task reasoning based on image content.

[0003] Existing autonomous driving systems typically acquire environmental images through multiple onboard cameras deployed around the vehicle, forming multiple initial observation images covering the vehicle's surroundings. These initial observation images can come from front-view cameras, side-view cameras, rear-view cameras, surround-view cameras, or multiple initial virtual cameras preset in a simulated environment. In a common implementation, cameras can be positioned in front of the vehicle, to the left front, right front, rear, left rear, and right rear, forming six initial observation images around the vehicle. This type of six-view image can expand the visual coverage around the vehicle to a certain extent, providing a more complete scene observation capability compared to a single front-view camera.

[0004] However, even with multiple initial observation images of the vehicle's surroundings, visual information may still be insufficient. On one hand, the position and orientation of the onboard camera or initial virtual camera are usually fixed in advance, making it impossible to actively adjust the observation position based on the visual language model's current inference needs. On the other hand, complex traffic scenarios may still present issues such as occluded areas, blind spots, targets that are too small, unclear relationships between side and rear targets, and difficulty in observing high-risk areas from appropriate angles. For example, in scenarios such as vehicle entry, intersection passage, lateral parallel traffic, pedestrian obstruction, and meeting on narrow roads, even if the system has acquired six images of the vehicle's surroundings, the visual language model may still be unable to accurately determine whether there are pedestrians behind an occluded area, the true spatial relationship between a vehicle to the side and the vehicle itself, or whether there are potential risks in a particular local area.

[0005] While visual language models can jointly understand six input images, they cannot change the image source, nor can they proactively "change angles" to view the target area when observation is insufficient, unlike human drivers. Existing methods typically only passively analyze given images, unable to actively request new observation positions and directions based on the current scene understanding. Consequently, the spatial understanding capabilities of visual language models in autonomous driving simulation scenarios remain limited by the initial fixed viewpoint.

[0006] On the other hand, autonomous driving simulation environments provide the technological foundation for generating virtual camera perspectives. Autonomous driving simulation platforms, exemplified by CARLA, are built on Unreal Engine and can provide simulation elements such as roads, vehicles, pedestrians, traffic facilities, ambient lighting, and sensors, while supporting the placement of virtual cameras within the simulation space. Unlike real vehicle-mounted cameras, virtual cameras in simulation environments are not limited by physical installation locations and can adjust their position, orientation, height, distance, and field of view according to task requirements, thereby generating supplementary images from different observation angles.

[0007] However, existing virtual camera usage methods mostly rely on preset fixed viewpoints for acquisition or manual specification of camera positions, lacking an active viewpoint generation mechanism tailored to the information acquisition needs of visual language models. In other words, current methods typically only pre-set several camera viewpoints or allow manual control of viewpoint switching, failing to automatically generate new observation areas, directions, and virtual camera poses based on the visual language model's assessment of the information sufficiency of multiple initial observation images. Consequently, while virtual cameras possess flexible acquisition capabilities, they do not form an effective closed loop with the visual language model's understanding process.

[0008] Furthermore, in autonomous driving simulation environments, if the position and orientation of virtual cameras are set directly based on the simulation world coordinate system, the semantic meaning of the virtual camera's viewpoint can easily drift when the vehicle moves, turns, changes lanes, or makes a U-turn. For example, a fixed right-hand direction in the world coordinate system may not always correspond to the right-hand region of the vehicle. Without a pose reference stably associated with the vehicle or onboard camera, it is difficult to guarantee spatial consistency and interpretability during multiple rounds of virtual viewpoint generation.

[0009] Therefore, there is an urgent need for a virtual camera viewpoint generation method and system for autonomous driving simulation. Based on six initial images around the vehicle, a visual language model is used to determine whether the current image information is sufficient, and a structured virtual viewpoint requirement is generated when the information is insufficient. Furthermore, the virtual viewpoint requirement is converted into a virtual camera target pose through camera pose anchor points and viewpoint parameter templates, and the autonomous driving simulation environment is controlled to generate supplementary viewpoint images and feed them back to the visual language model, thereby improving the spatial understanding ability and information acquisition completeness in the autonomous driving simulation scenario. Summary of the Invention

[0010] The purpose of this invention is to provide a virtual camera perspective generation method and system for autonomous driving simulation. Based on multiple initial observation images, the visual language model actively requests supplementary observation perspectives according to the current image understanding results, and generates corresponding virtual camera images in the autonomous driving simulation environment. This overcomes the problems of fixed initial perspective limitations, insufficient information in multi-view initial images, visual language models only being able to passively receive images, lack of cognitive demand drive for virtual cameras, and insufficient consistency in perspective generation space in the prior art.

[0011] To achieve the above objectives, the present invention provides the following solution: A method for generating virtual camera viewpoints for autonomous driving simulation includes the following steps: S1: Acquire the vehicle's operating status, scene status information, and multiple initial observation images in the autonomous driving simulation environment; S2, organize the multiple initial observation images according to their corresponding viewing angle, image source, or acquisition time to form an initial observation image set, and input the initial observation image set into the visual language model; S3. Based on the visual language model, perform joint scene understanding on the initial observed image set and determine whether the current image information meets the task requirements; if not, generate structured virtual perspective requirements from the visual language model. S4. Establish camera pose anchor points based on the vehicle pose or the pose of the vehicle camera, and match preset view parameter templates according to the structured virtual view requirements to generate view parameters relative to the camera pose anchor points. S5 calculates the target pose of the virtual camera based on the camera pose anchor point and view parameters. After spatial constraint check, it controls the virtual camera to acquire supplementary view images and feeds them back to the visual language model to form a closed loop iteration until the current image information meets the task requirements or reaches the preset maximum number of iterations, and finally outputs the scene understanding result.

[0012] Furthermore, in S1, the autonomous driving simulation environment is the CARLA autonomous driving simulation environment built on Unreal Engine. The autonomous driving simulation environment is used to provide a renderable 3D traffic scene and supports the control of the position, attitude, field of view and image acquisition parameters of the virtual camera.

[0013] Furthermore, in S1, the vehicle's operating state includes the vehicle's position, attitude, orientation, speed, steering state, and vehicle control state in the simulated world coordinate system, with attitude encompassing roll angle, pitch angle, and yaw angle. The scene status information includes road topology, lane structure, road boundaries, surrounding vehicles, pedestrians, static obstacles, traffic lights, traffic signs, and target observation area information; The multiple initial observation images are acquired by an onboard camera, a surround-view camera, a simulated virtual camera, or a multi-sensor image generation module; the multiple initial observation images are six-view initial observation images, including a front image, a left front image, a right front image, a rear image, a left rear image, and a right rear image; each initial observation image has a corresponding view orientation label, which is used to indicate the observation direction of the image relative to the vehicle.

[0014] Furthermore, S3 also includes: if the current image information meets the task requirements, then output the scene understanding result; When the visual language model determines whether the current image information meets the task requirements, it includes at least one of the following judgment criteria: Determine whether the target observation area is within the visible range of the initial observation image set; Determine whether the target object is fully visible in the image; Determine whether the target object is obscured by vehicles, pedestrians, obstacles, or road structures; Determine whether the relative orientation and spatial distance between the target object and the vehicle can be determined; Determine whether the current image lacks information about side, rear, top, or local target regions; Determine whether the scale of the target object in the current image is lower than a preset analyzable scale threshold; If at least one of the judgment criteria does not meet the preset conditions, the current image information is determined not to meet the task requirements.

[0015] Furthermore, in S3, the structured virtual perspective requirement includes at least one of the following: observation area field, observation object field, perspective type field, orientation category field, information missing reason field, and observation priority field; The observation area field is used to represent the spatial range to be observed. The observed object field is used to represent vehicles, pedestrians, obstacles, lane boundaries, or other traffic participants; The viewpoint type field is used to represent side view, rear view, top view, oblique view, or local target view. The orientation category field is used to indicate the position of the observation area relative to the front, rear, left, right, left front, right front, left rear, or right rear of the vehicle. The missing information reason field is used to indicate occlusion, blind spot, too small scale, unclear spatial relationship or incomplete target; The observation priority field is used to indicate the importance of supplementary observations.

[0016] Furthermore, in S4, camera pose anchor points are established based on the vehicle's pose or the pose of the onboard camera, specifically including: Camera pose anchor points are divided into vehicle pose anchor points and vehicle-mounted camera pose anchor points; When the vehicle pose is used as the camera pose anchor point, the virtual camera target pose is determined relative to the vehicle coordinate system and used to generate supplementary view images from the side, rear, oblique or top views around the vehicle. When the pose of the vehicle-mounted camera is used as the camera pose anchor point, the virtual camera target pose is determined relative to the vehicle-mounted camera coordinate system, which is used to perform deflection, elevation, lateral shift or local magnification observation based on the existing initial image viewpoint. The camera pose anchor point serves as a spatial reference benchmark for generating the virtual camera viewpoint and is updated in real time according to the vehicle's operating status.

[0017] Furthermore, in S4, the preset viewing angle parameter template includes at least one of the following: lateral observation template, rearward observation template, oblique observation template, top-down observation template, and local target observation template; The viewpoint parameters include relative pose parameters and field of view parameters; The relative pose parameters include lateral offset, longitudinal offset, height offset, azimuth adjustment, pitch adjustment, and roll adjustment. The field of view parameter is used to represent the field of view range of the virtual camera; The relative pose parameters and the field of view parameters are used together to determine the position offset, attitude adjustment and field of view range of the virtual camera relative to the camera pose anchor point.

[0018] Furthermore, in S5, the calculation of the target pose of the virtual camera based on the camera pose anchor point and viewpoint parameters specifically includes: The field of view of the virtual camera is obtained based on the field of view angle parameter; Based on the camera pose anchor point and relative pose parameters, relative pose combination or coordinate transformation is performed to obtain the target position and target orientation of the virtual camera in the autonomous driving simulation environment, and spatial constraint checks are performed on the target position and target orientation. The spatial constraint check includes: determining whether the target location is within the renderable area, determining whether the target location has a spatial conflict with vehicles, pedestrians, obstacles or road structures, determining whether the target orientation can cover the target observation area, and determining whether there is severe occlusion between the virtual camera and the observed target; when the preset spatial constraint conditions are not met, the virtual camera target position, height, distance or orientation is corrected.

[0019] Furthermore, in S5, the virtual camera is controlled to acquire supplementary viewpoint images and feed them back to the visual language model, forming a closed-loop iteration until the current image information meets the task requirements or reaches the preset maximum number of iterations, and finally outputs the scene understanding result, specifically including: After satisfying the preset spatial constraints, the virtual camera in the autonomous driving simulation environment is controlled to acquire supplementary viewpoint images, and the supplementary viewpoint images are fed back to the visual language model, so that the visual language model can combine the initial observation image set and the supplementary viewpoint images to continue to perform scene understanding and information sufficiency judgment. If the current image information still does not meet the task requirements, repeat steps S3-S5 to generate structured virtual viewpoint requirements, generate viewpoint parameters, calculate virtual camera target pose, acquire supplementary viewpoint images, and provide image feedback, until the visual language model determines that it has obtained image information that meets the task requirements, or reaches the preset maximum number of iterations, and finally outputs the scene understanding result.

[0020] This invention also provides a virtual camera viewpoint generation system for autonomous driving simulation, applied to the above-described virtual camera viewpoint generation method for autonomous driving simulation, comprising: The simulation state acquisition module is used to acquire the vehicle's operating state, scene state information, and multiple initial observation images in the autonomous driving simulation environment. The initial image input module is used to organize the multiple initial observation images according to their corresponding viewpoint, image source, or acquisition time to form an initial observation image set, and input the initial observation image set into a pre-constructed visual language model; The information sufficiency determination module is used to perform joint scene understanding on the initial observation image set based on the visual language model, and to determine whether the current image information meets the task requirements. The viewpoint requirement parsing module is used to generate and parse structured virtual viewpoint requirements when the current image information does not meet the task requirements. The pose anchor point establishment module is used to establish camera pose anchor points based on the vehicle pose or the pose of the vehicle camera. The viewpoint parameter generation module is used to match a preset viewpoint parameter template according to the structured virtual viewpoint requirements and generate viewpoint parameters relative to the camera pose anchor point. The virtual camera pose calculation module is used to calculate the target pose of the virtual camera based on the camera pose anchor points and view parameters; The pose constraint correction module is used to check the spatial constraint of the virtual camera target pose and correct it when the preset spatial constraint conditions are not met. The virtual camera rendering control module is used to control the virtual camera in the autonomous driving simulation environment to acquire supplementary viewpoint images; The image feedback and recording module is used to feed back the supplementary viewpoint image to the visual language model and record the generation information corresponding to the supplementary viewpoint image; The closed-loop iterative management module is used to repeatedly execute the processes of information sufficiency determination, viewpoint requirement analysis, viewpoint parameter generation, virtual camera pose calculation, supplementary viewpoint image acquisition, and image feedback until the visual language model determines that it has obtained image information that meets the task requirements, or reaches the preset maximum number of iterations.

[0021] According to specific embodiments provided by the present invention, the technical effects of the virtual camera perspective generation method and system for autonomous driving simulation provided by the present invention are mainly reflected in: (1) Active supplementary observation based on multiple perspectives: This invention does not rely solely on a single image, but rather makes a sufficiency judgment based on multiple initial observation images; when the multiple initial observation images still cannot meet the task requirements, it actively generates supplementary virtual perspectives to improve the image information coverage capability.

[0022] (2) From passive image analysis to active perspective acquisition: By combining the information sufficiency judgment result of the visual language model with the virtual camera control process, the visual language model can actively request new observation perspectives based on the current image understanding results, thus realizing the transformation from passively receiving images to actively acquiring supplementary images.

[0023] (3) Structured perspective requirement conversion: By converting the observation requirements output by the visual language model into structured virtual perspective requirements, and further generating executable perspective parameters based on the preset perspective parameter template, the reliability of the conversion between natural language observation requirements and virtual camera control parameters is improved.

[0024] (4) Maintaining consistency of view space based on camera pose anchor point: This invention defines the virtual camera target pose as a relative pose relative to the vehicle pose or the vehicle camera pose, rather than directly fixed in the simulation world coordinate system, so that the view semantics such as "left, right, rear, right front" can be updated in real time with the movement of the vehicle, thereby avoiding semantic drift of the supplementary view when the vehicle turns, changes lanes or makes a U-turn.

[0025] (5) Closed-loop completion of scene information: Through the closed-loop mechanism of "multi-view initial image input - model understanding and judgment - structured view requirement generation - view template matching - virtual camera pose calculation - supplementary image feedback", the visual language model can gradually complete the visual information of occluded areas, blind areas and local target areas, thereby improving the spatial understanding ability and information acquisition completeness in autonomous driving simulation scenarios.

[0026] In summary, this invention achieves active virtual camera perspective generation for autonomous driving simulation by multi-view initial image input, information sufficiency determination, structured virtual perspective requirement generation, camera pose anchor point establishment, perspective parameter template mapping, virtual camera pose calculation, spatial constraint correction, and supplementary perspective image feedback. This improves the completeness of information acquisition and the reliability of scene understanding of the visual language model in the autonomous driving simulation environment. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of a virtual camera viewpoint generation method for autonomous driving simulation provided in an embodiment of the present invention. Figure 2 A schematic diagram of the overall framework of a virtual camera perspective generation system for autonomous driving simulation provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the initial observation from six perspectives provided in an embodiment of the present invention.

[0029] Figure 4 This is a schematic diagram illustrating the active supplementary virtual perspective generation provided in an embodiment of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] This invention provides a method and system for generating virtual camera perspectives for autonomous driving simulation. This method is primarily aimed at applications such as autonomous driving simulation testing, visual language model scene understanding verification, virtual sensor perspective generation, and auxiliary analysis of complex traffic scenarios, and is particularly suitable for CARLA-based autonomous driving simulation environments. The method uses multi-view initial images as basic observation information, and uses a visual language model to determine whether the current image information meets the task requirements. When the initial image still has problems such as occlusion, blind spots, excessively small target scale, or unclear spatial relationships, a structured virtual perspective requirement is generated, and supplementary virtual camera perspective images are generated in the simulation space based on camera pose anchor points and perspective parameter templates. The technical chain of this method includes "multi-view initial image input—information sufficiency determination—structured virtual perspective requirement generation—camera pose anchor point establishment—perspective parameter template matching—virtual camera pose calculation—spatial constraint correction—supplementary perspective image feedback—closed-loop iteration," and its technical route is as follows: Figure 1 As shown.

[0033] like Figure 1 As shown, the virtual camera viewpoint generation method for autonomous driving simulation provided in this embodiment of the invention includes the following steps: S1: Acquire the vehicle's operating status, scene status information, and multiple initial observation images in the autonomous driving simulation environment; S2, organize the multiple initial observation images according to their corresponding viewing angle, image source, or acquisition time to form an initial observation image set, and input the initial observation image set into the visual language model; In this embodiment, the visual language model can be an existing pre-trained multimodal visual language model with image understanding, text generation, and multi-image joint analysis capabilities, or it can be a visual language model fine-tuned based on autonomous driving simulation images, traffic scene description text, and scene question-and-answer data. This invention does not limit the specific network structure and training method of the visual language model. The visual language model is used to perform semantic understanding, spatial relationship analysis, and information sufficiency judgment on the initial observation image set and supplementary viewpoint images.

[0034] S3. Based on the visual language model, perform joint scene understanding on the initial observed image set and determine whether the current image information meets the task requirements; if not, generate structured virtual perspective requirements from the visual language model. S4. Establish camera pose anchor points based on the vehicle pose or the pose of the vehicle camera, and match preset view parameter templates according to the structured virtual view requirements to generate view parameters relative to the camera pose anchor points. S5 calculates the target pose of the virtual camera based on the camera pose anchor point and view parameters. After spatial constraint check, it controls the virtual camera to acquire supplementary view images and feeds them back to the visual language model to form a closed loop iteration until the current image information meets the task requirements or reaches the preset maximum number of iterations, and finally outputs the scene understanding result.

[0035] The steps of each embodiment of the present invention are described in detail below: I. Regarding step S1: In step S1, the vehicle's operating status, scene status information, and multiple initial observation images in the autonomous driving simulation environment are first acquired.

[0036] The autonomous driving simulation environment can be the CARLA autonomous driving simulation environment built on Unreal Engine, or other simulation platforms capable of providing 3D traffic scene rendering and virtual camera control. This autonomous driving simulation environment provides renderable 3D traffic scenes and supports control over the virtual camera's position, pose, field of view, and image acquisition parameters. The principle of this method is as follows: Figure 2 As shown.

[0037] The vehicle's operating status includes its position, attitude, orientation, speed, steering status, and vehicle control status in the simulated world coordinate system. Attitude can include roll angle, pitch angle, and yaw angle, while orientation can be represented by heading angle. Scene state information includes road topology, lane structure, road boundaries, surrounding vehicles, pedestrians, static obstacles, traffic lights, traffic signs, and target observation area information.

[0038] The multiple initial observation images are multi-view images surrounding the vehicle or covering the target traffic area. They can be obtained from onboard cameras, surround-view cameras, simulation virtual cameras, or multi-sensor image generation modules. The number of images can be set according to the vehicle sensor configuration, simulation task requirements, or observation area range; for example, it can be four, six, eight, or other numbers. Each initial observation image has a corresponding view orientation label to indicate the observation direction of the image relative to the vehicle or target traffic area.

[0039] In a preferred embodiment, the multiple initial observation images are six-view images, including a frontal image, a left frontal image, a right frontal image, a rearal image, a left rearal image, and a right rearal image. Through these six-view images, the visual language model can obtain basic visual information about the forward, lateral, and rearward regions surrounding the vehicle, providing initial observational basis for subsequent information sufficiency judgment and supplementary virtual perspective generation. The six-view initial observation relationships are as follows: Figure 3 As shown.

[0040] In one specific implementation, the system obtains the vehicle's Transform information through the vehicle state interface provided by CARLA. This Transform information includes Location and Rotation, where Location represents the vehicle's spatial position in the simulation world coordinate system, and Rotation represents the vehicle's attitude angle. The system can also obtain information about surrounding traffic participants and road environment elements through the simulation environment interface, providing a scene basis for subsequent virtual camera target pose calculations.

[0041] II. Regarding step S2: In step S2, the multiple initial observation images are organized according to their corresponding viewpoints or image sources to form an initial observation image set, and the initial observation image set is input into the visual language model.

[0042] Specifically, each initial observation image can be assigned a corresponding viewpoint identifier based on its acquisition location, image source, and acquisition time, and then assembled into an initial observation image set according to a preset order. The viewpoint identifier can include front, left front, right front, rear, left rear, right rear, left side, right side, top view, or other viewpoint categories.

[0043] In one specific embodiment, six initial images are used as the initial observation image set, labeled as front view, left front view, right front view, rear view, left rear view, and right rear view, respectively. These six images and their view labels are then input into the visual language model. After receiving the initial observation image set, the visual language model performs a joint analysis of the road structure, vehicles, pedestrians, obstacles, traffic signs, lane relationships, and spatial relationships within the images.

[0044] By using multiple initial observation images as the basic input, instead of just a single fixed-view image, the visual language model can obtain more complete basic environmental information. If the initial observation image set already meets the task requirements, there is no need to generate additional supplementary virtual views; if the initial observation image set still has missing information, the process of generating supplementary virtual views is triggered.

[0045] III. Regarding step S3: Step S3 can be divided into two steps: S301, Information Sufficiency Determination: In S301, the visual language model performs joint scene understanding based on the initial set of observed images and determines whether the current image information meets the task requirements. The task requirements include at least one of the following: autonomous driving scene understanding, traffic participant relationship analysis, occluded area observation, target vehicle location confirmation, road traffic status judgment, local risk area identification, and driver assistance decision analysis.

[0046] In one specific implementation, whether the current image information meets the task requirements can be determined jointly by the semantic judgment result of the visual language model and the target location, target category, target bounding box, depth information, or semantic segmentation information provided by the simulation environment. The system can determine whether the current image information meets the task requirements based on whether the target is within the field of view of any initial observation image, whether the target detection box is complete, the proportion of the target's visible area, the relative orientation and distance between the target and the vehicle, and whether the target's observation area is occluded by other vehicles, pedestrians, obstacles, or road structures.

[0047] When the visual language model determines whether the current image information meets the task requirements, it includes at least one of the following judgment criteria: Determine whether the target observation area is within the visible range of the initial observation image set; Determine whether the target object is fully visible in the image; Determine whether the target object is obscured by vehicles, pedestrians, obstacles, or road structures; Determine whether the relative orientation and spatial distance between the target object and the vehicle can be determined; Determine whether the current image lacks information about side, rear, top, or local target regions; Determine whether the scale of the target object in the current image is lower than a preset analyzable scale threshold; If at least one of the judgment criteria does not meet the preset conditions, the current image information is determined not to meet the task requirements.

[0048] Specifically, the visual language model first identifies the visible road structure, traffic participants, obstacles, lane relationships, and target areas in the six initial images, and then determines whether there is any missing information in the current six images. This missing information includes situations such as the target observation area not being visible in the six images, the target object being occluded, the target object being too small, the spatial relationship between the target and the vehicle being unclear, insufficient information on side or rear blind spots, and local risk areas not being observable from a suitable angle.

[0049] It should be noted that the visual language model's recognition of road structure, traffic participants, obstacles, lane relationships, and target areas can be achieved based on the existing visual language model's image semantic understanding and multi-image joint analysis capabilities. The improvement of this invention does not focus on changing the internal network structure of the visual language model, but rather on using the information sufficiency judgment results of the visual language model to drive the generation of virtual camera viewpoint requirements, pose calculation, and supplementary image feedback, thereby forming an active supplementary observation closed loop.

[0050] The preset analyzable scale threshold may include a threshold for the proportion of the target detection box pixel area to the total image area, a threshold for the shorter side pixels of the target detection box, or a threshold for the proportion of the visible area of ​​the target. For example, when the proportion of the target detection box pixel area to the total image area is less than a preset proportion threshold, or the shorter side pixels of the target detection box are less than a preset pixel threshold, or the proportion of the visible area of ​​the target is less than a preset visible proportion threshold, it can be determined that the target scale is too small or the target is not fully displayed. These thresholds can be preset according to the simulation task, image resolution, and target category.

[0051] For example, if the initial six images can fully display the vehicles, pedestrians, and road structures around the vehicle and support the current task judgment, the visual language model determines that the current image information meets the task requirements. If a target in the initial image set is occluded, a lateral region is incomplete, or a local risk area cannot be determined, the visual language model determines that the current image information does not meet the task requirements.

[0052] S302, Viewpoint Requirements Analysis: In step S302, when the current image information does not meet the task requirements, a structured virtual perspective requirement is generated by the visual language model. This structured virtual perspective requirement represents the region, object, direction, perspective type, and reasons for information loss that the visual language model wishes to observe further, thereby converting the model's semantic understanding results into a structured input that the virtual camera control process can parse.

[0053] Specifically, the structured virtual perspective requirement includes at least one of the following: observation area field, observation object field, perspective type field, orientation category field, information missing reason field, and observation priority field. The observation area field indicates the spatial range to be supplemented for observation; the observation object field indicates vehicles, pedestrians, obstacles, lane boundaries, or other traffic participants; the perspective type field indicates side-view, rear-view, top-view, oblique view, or local target observation perspective; the orientation category field indicates the position of the observation area relative to the front, rear, left, right, left front, right front, left rear, or right rear of the vehicle; the information missing reason field indicates occlusion, blind spots, insufficient scale, unclear spatial relationships, or incomplete targets; and the observation priority field indicates the importance of supplementing the observation.

[0054] In one implementation, the structured virtual viewpoint requirement can be represented using fielded text, key-value pairs, or a structured data format. For example, the structured virtual viewpoint requirement might include: the observation area is the right-front occluded area, the observed object is the area behind the occluded vehicle, the viewpoint type is an oblique observation viewpoint, the orientation category is right-front, the reason for missing information is occlusion, and the observation priority is high. The system parses the structured virtual viewpoint requirement, extracts the target observation area, viewpoint type, and orientation category, and uses these as inputs for subsequent viewpoint parameter template matching and virtual camera pose calculation.

[0055] When the structured virtual viewpoint requirements output by the visual language model have missing fields, unclear orientation categories, or cannot locate the observed object, the system can complete them based on the current task type, target area location, and preset default viewpoint template, or request the visual language model to output structured virtual viewpoint requirements that meet the preset format requirements again. This improves the stability and executability of the conversion between the semantic requirements of the visual language model and the virtual camera control parameters.

[0056] IV. Regarding step S4: Step S4 can be divided into two steps: S401, Pose Anchor Point Establishment: In step S401, camera pose anchor points are established based on the vehicle's pose or the pose of the onboard camera. These camera pose anchor points serve as a spatial reference for generating the virtual camera's viewpoint and are updated in real-time according to the vehicle's operating status.

[0057] In one specific implementation, the system uses the vehicle's pose as the camera pose anchor point. In this case, the virtual camera target pose is determined relative to the vehicle's coordinate system, used to generate supplementary view images from the front, rear, left, right, oblique, or top-down perspectives around the vehicle. For example, when generating a supplementary right-side view image, the virtual camera is always located in a preset area on the right side of the vehicle, and will not deviate from the "right side of the vehicle" perspective semantics due to the vehicle turning, changing lanes, or making a U-turn in the simulated world coordinate system.

[0058] In another specific implementation, the system uses the pose of the vehicle-mounted camera or the initial virtual camera pose as the camera pose anchor point. In this case, the virtual camera target pose is determined relative to the corresponding camera coordinate system, and is used to perform deflection, elevation, lateral movement, zoom in, or local target magnification observation based on the existing initial observation image.

[0059] Since the virtual camera target pose is determined relative to the camera pose anchor point, rather than being directly fixed to the simulated world coordinate system, the supplementary viewpoint can still maintain semantic consistency in orientation relative to the vehicle or the onboard camera when the vehicle moves, turns, changes lanes, or makes a U-turn. By establishing camera pose anchor points, semantic drift in viewpoints during multiple rounds of virtual viewpoint generation can be avoided, improving the spatial consistency and interpretability of the supplementary viewpoint image.

[0060] S402, Viewpoint Parameter Generation: In step S402, a preset view parameter template is matched according to the structured virtual view requirements, and view parameters relative to the camera pose anchor point are generated.

[0061] The viewpoint parameters include relative pose parameters and field of view parameters. The relative pose parameters can be expressed as: in, This represents the relative pose parameters generated by the structured virtual viewpoint requirements. Indicates the lateral offset. Indicates the vertical offset. Indicates the height offset. Indicates the azimuth adjustment amount. Indicates the pitch angle adjustment amount. This indicates the roll angle adjustment amount. The field of view parameter... The field of view is used to represent the virtual camera's field of view. The relative pose parameter and the field of view parameter are used together to determine the virtual camera's position offset, attitude adjustment, and field of view range relative to the camera pose anchor point.

[0062] Specifically, the corresponding view parameter template is first determined based on the view type field in the structured virtual view requirements. When the view type is side view, the side view template is matched; when the view type is rear view, the rear view template is matched; when the view type is oblique view, the oblique view template is matched; when the view type is top view, the top view template is matched; and when the view type is local target view, the local target view template is matched.

[0063] For lateral observation templates, the system generates lateral offset, azimuth adjustment, and observation target based on the left or right azimuth category; for rearward observation templates, the system generates longitudinal offset, azimuth adjustment, and observation target based on the rear target area; for oblique observation templates, the system generates lateral offset, longitudinal offset, and azimuth adjustment based on the left front, right front, left rear, or right rear azimuth category; for top-down observation templates, the system generates altitude offset and pitch adjustment; for local target observation templates, the system generates observation distance, field of view parameters, and the center point of the observation target based on the target center position.

[0064] By using viewpoint parameter templates, the structured virtual viewpoint requirements output by the visual language model can be converted into explicit and executable virtual camera control parameters, avoiding the problems of unstable or unreproducible viewpoint control caused by relying solely on natural language descriptions.

[0065] V. Regarding step S5: In step S5, relative pose combination or coordinate transformation is performed based on the camera pose anchor point and the relative pose parameters to obtain the target position and target orientation of the virtual camera in the autonomous driving simulation environment, and spatial constraint checks are performed on the target position and target orientation. After the preset spatial constraint conditions are met, the virtual camera is controlled to acquire supplementary view images, and the supplementary view images are fed back to the visual language model, so that the visual language model can continue to perform scene understanding and information sufficiency judgment by combining the initial observation image set and the supplementary view images. If the current image information still does not meet the task requirements, the structured virtual view requirement generation, view parameter generation, virtual camera target pose calculation, supplementary view image acquisition and image feedback process is repeated until the visual language model judges that it has obtained image information that meets the task requirements, or the preset maximum number of iterations is reached.

[0066] Specifically, step S5 can be divided into the following steps: S501, Virtual Camera Pose Calculation: In step S501, relative pose combination or coordinate transformation is performed based on camera pose anchor points and relative pose parameters to obtain the target position and target orientation of the virtual camera in the autonomous driving simulation environment.

[0067] Specifically, the system first determines the relative position and attitude of the virtual camera in the camera pose anchor point coordinate system. Then, based on the position and attitude of the camera pose anchor point in the simulation world coordinate system, the system transforms the relative position and attitude to the simulation world coordinate system. Subsequently, the system calculates the virtual camera target orientation based on the virtual camera target position and the observation target position, enabling the virtual camera to be aligned with the target observation area.

[0068] The virtual camera target pose can be determined by the camera pose anchor point and the relative pose parameters, and their relationship is as follows: in, This represents the virtual camera target pose. Indicates the camera pose anchor point. This represents the relative pose parameters with respect to the camera pose anchor point. This indicates an operation that combines relative poses or transforms coordinates based on pose anchor points.

[0069] S502, Pose Constraint Correction: Step S502: Perform a spatial constraint check on the target position and target orientation.

[0070] Specifically, the system determines whether the virtual camera target location is within the renderable area, whether the virtual camera target location spatially conflicts with vehicles, pedestrians, obstacles, or road structures, whether the virtual camera target orientation can cover the target observation area, and whether there is severe occlusion between the virtual camera and the observed target. When the virtual camera target location or orientation does not meet the preset spatial constraints, the system corrects the virtual camera's height, distance, lateral offset, longitudinal offset, or orientation until the preset spatial constraints are met.

[0071] S503, Virtual Camera Rendering Control: In step S503, after the preset spatial constraints are met, the system controls the virtual camera in the autonomous driving simulation environment to acquire supplementary viewpoint images and feeds the supplementary viewpoint images back to the visual language model, so that the visual language model can continue to perform scene understanding and information sufficiency judgment by combining the initial observation image set and the supplementary viewpoint images.

[0072] In one specific implementation, the system calls the virtual camera sensor interface in CARLA to set the virtual camera to the target position and render the image according to the target orientation. The supplementary view images can be images from the right side of the vehicle, the left side of the vehicle, the rear of the vehicle, the right front oblique image, the left rear oblique image, a top-down view, or images of a local target area.

[0073] S504, Image Feedback and Recording: While providing supplementary viewpoint images, the system records the corresponding image number, viewpoint name, camera pose anchor point, virtual camera target pose, viewpoint parameters, observed target, reason for missing information, and feedback round information. The image number distinguishes different acquired images; the viewpoint name identifies right-side view, rear view, top-down view, oblique view, or local observation view; and the feedback round records which round of the visual language model request result the image belongs to.

[0074] S505, Closed-Loop Iterative Management: If the visual language model, after combining the initial observation image set and supplementary viewpoint images, still determines that the current image information does not meet the task requirements, the system will repeatedly execute the process of generating structured virtual viewpoint requirements, matching viewpoint parameter templates, calculating virtual camera target pose, checking spatial constraints, acquiring supplementary viewpoint images, and providing image feedback, until the visual language model determines that it has obtained image information that meets the task requirements, or reaches the preset maximum number of iterations.

[0075] like Figure 2As shown, this embodiment of the invention establishes a closed-loop interaction process between multi-view initial observation images, a visual language model, and a virtual camera. The visual language model first performs joint understanding based on multiple initial observation images; in one specific embodiment, the multiple initial observation images are six-view images of the area around the vehicle, respectively covering the areas in front, left front, right front, rear, left rear, and right rear. When the six-view images can meet the current task requirements, the visual language model directly outputs the scene understanding result; when the six-view images still have problems such as occluded areas, blind spots, targets that are too small, or unclear spatial relationships, the visual language model generates structured virtual view requirements.

[0076] The system matches corresponding viewpoint parameter templates based on the structured virtual viewpoint requirements and calculates the virtual camera target pose using the vehicle's pose or the onboard camera's pose as camera pose anchor points. Subsequently, the system controls the virtual camera to acquire supplementary viewpoint images in the CARLA autonomous driving simulation environment and feeds these supplementary viewpoint images back to the visual language model. The visual language model combines the initial observation image set and the supplementary viewpoint images to continue scene understanding and information sufficiency assessment. This process can be repeated until the visual language model obtains image information that meets the task requirements, or reaches a preset maximum number of iterations. The effect of actively supplementing the virtual viewpoint generation is as follows: Figure 4 As shown.

[0077] For example, during the autonomous vehicle's journey along the park's roads, the system has acquired initial images from six perspectives surrounding the vehicle, covering the areas in front, to the left front, to the right front, behind, to the left rear, and to the right rear. However, a parked vehicle in the right front image obscures the road edge, and the visual language model cannot determine whether a pedestrian is behind the obscured vehicle. At this point, the visual language model generates a structured virtual viewpoint requirement, where the observation area is the obscured right front area, the observation object is the area behind the obscured vehicle, the viewpoint type is an oblique observation viewpoint, the orientation category is right front, and the reason for the missing information is obstruction. Using the vehicle's pose as the camera pose anchor point, the system calls the oblique observation template to generate corresponding lateral offset, longitudinal offset, height offset, and azimuth adjustment, and calculates the virtual camera target pose. Subsequently, in the CARLA autonomous driving simulation environment, the system controls the virtual camera to move to a preset position to the right front, allowing it to capture supplementary viewpoint images facing the area behind the obscured vehicle, and feeds these supplementary viewpoint images back to the visual language model. Based on the supplementary viewpoint images, the visual language model continues to determine whether there are pedestrians or other traffic participants within the obscured area.

[0078] Through the aforementioned closed-loop iterative approach, the visual language model can gradually supplement the visual information of occluded areas, blind spots, and local target areas based on its own information needs, thereby breaking through the limitations of fixed initial viewpoint input and improving the spatial understanding ability of complex autonomous driving simulation scenarios.

[0079] In another embodiment, the present invention also provides a virtual camera viewpoint generation system for autonomous driving simulation, applied to performing the above-described virtual camera viewpoint generation method for autonomous driving simulation, comprising: The simulation state acquisition module is used to acquire the vehicle's operating state and scene state information in the autonomous driving simulation environment; The initial image input module is used to acquire multiple initial observation images and organize them into an initial observation image set according to the corresponding viewpoint or image source. The information sufficiency determination module is used to perform scene understanding on the initial observation image set or supplementary viewpoint images through a visual language model, and to determine whether the current image information meets the task requirements. The viewpoint requirement analysis module is used to generate structured virtual viewpoint requirements when the current image information does not meet the task requirements. The pose anchor point establishment module is used to establish camera pose anchor points based on the vehicle pose or the pose of the on-board camera, and to update them in real time according to the vehicle's running status. The viewpoint parameter generation module is used to match a preset viewpoint parameter template according to the structured virtual viewpoint requirements and generate viewpoint parameters relative to the camera pose anchor point. The virtual camera pose calculation module is used to calculate the virtual camera target pose based on the camera pose anchor points and view parameters; The pose constraint correction module is used to check the spatial constraint of the virtual camera target pose and correct it when the preset spatial constraint conditions are not met. The virtual camera rendering control module is used to control the virtual camera in the autonomous driving simulation environment to acquire supplementary viewpoint images; The image feedback and recording module is used to feed back the supplementary viewpoint image to the visual language model and record the generation information corresponding to the supplementary viewpoint image; The closed-loop iterative management module is used to repeatedly execute the processes of information sufficiency determination, viewpoint requirement analysis, viewpoint parameter generation, virtual camera pose calculation, and image feedback until the visual language model determines that it has obtained image information that meets the task requirements, or reaches the preset maximum number of iterations.

[0080] Through the above methods, embodiments of the present invention can determine whether the current image information meets the scene understanding requirements of the visual language model based on multiple initial observation images, and generate supplementary virtual perspective images using camera pose anchor points, structured virtual perspective requirements, and perspective parameter templates when information is insufficient. In one specific embodiment, the multiple initial observation images can be six-view images around the vehicle; compared to relying solely on fixed initial perspective images, the present invention can proactively generate new supplementary observation perspectives according to the information requirements of the visual language model, effectively expanding the observation range of the visual language model and improving spatial understanding capabilities and image information integrity in complex traffic scenarios.

[0081] The present invention also discloses an electronic device comprising one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more application programs are configured to perform the virtual camera view generation method for autonomous driving simulation as described above.

[0082] The present invention also discloses a non-transitory computer-readable storage medium storing a computer program thereon, characterized in that the computer program, when executed by a processor, implements the virtual camera view generation method for autonomous driving simulation as described above.

[0083] Those skilled in the art should understand that the technical solutions described in this invention are not limited to the specific embodiments described above.

[0084] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for generating virtual camera viewpoints for autonomous driving simulation, characterized in that, Includes the following steps: S1: Acquire the vehicle's operating status, scene status information, and multiple initial observation images in the autonomous driving simulation environment; S2, organize the multiple initial observation images according to their corresponding viewing angle, image source, or acquisition time to form an initial observation image set, and input the initial observation image set into the visual language model; S3, based on the visual language model, perform joint scene understanding on the initial observation image set and determine whether the current image information meets the task requirements; if not, the visual language model generates structured virtual perspective requirements; the structured virtual perspective requirements include at least one of the following: observation area field, observation object field, perspective type field, orientation category field, information missing reason field, and observation priority field; S4. Establish camera pose anchor points based on the vehicle pose or the pose of the vehicle-mounted camera, and match preset view parameter templates according to the structured virtual view requirements to generate view parameters relative to the camera pose anchor points; the preset view parameter templates include at least one of the following: lateral view template, rear view template, oblique view template, top view template, and local target view template. In S4, camera pose anchor points are established based on the vehicle's pose or the pose of the onboard camera, specifically including: Camera pose anchor points are divided into vehicle pose anchor points and vehicle-mounted camera pose anchor points; When the vehicle pose is used as the camera pose anchor point, the virtual camera target pose is determined relative to the vehicle coordinate system and used to generate supplementary view images from the side, rear, oblique or top views around the vehicle. When the pose of the vehicle-mounted camera is used as the camera pose anchor point, the virtual camera target pose is determined relative to the vehicle-mounted camera coordinate system, which is used to perform deflection, elevation, lateral shift or local magnification observation based on the existing initial image viewpoint. The camera pose anchor point serves as a spatial reference benchmark for generating the virtual camera viewpoint and is updated in real time according to the vehicle's operating status. S5 calculates the target pose of the virtual camera based on the camera pose anchor point and view parameters. After spatial constraint check, it controls the virtual camera to acquire supplementary view images and feeds them back to the visual language model to form a closed loop iteration until the current image information meets the task requirements or reaches the preset maximum number of iterations, and finally outputs the scene understanding result.

2. The virtual camera perspective generation method for autonomous driving simulation according to claim 1, characterized in that, In S1, the autonomous driving simulation environment is the CARLA autonomous driving simulation environment built on Unreal Engine. The autonomous driving simulation environment is used to provide a renderable 3D traffic scene and supports the control of the position, attitude, field of view and image acquisition parameters of the virtual camera.

3. The virtual camera perspective generation method for autonomous driving simulation according to claim 1, characterized in that, In S1, the vehicle's operating state includes the vehicle's position, attitude, orientation, speed, steering state, and vehicle control state in the simulated world coordinate system. The attitude includes roll angle, pitch angle, and yaw angle. The scene status information includes road topology, lane structure, road boundaries, surrounding vehicles, pedestrians, static obstacles, traffic lights, traffic signs, and target observation area information; The multiple initial observation images are acquired by an onboard camera, a surround-view camera, a simulated virtual camera, or a multi-sensor image generation module; the multiple initial observation images are six-view initial observation images, including a front image, a left front image, a right front image, a rear image, a left rear image, and a right rear image; each initial observation image has a corresponding view orientation label, which is used to indicate the observation direction of the image relative to the vehicle.

4. The virtual camera perspective generation method for autonomous driving simulation according to claim 1, characterized in that, S3 also includes: if the current image information meets the task requirements, then output the scene understanding result; When the visual language model determines whether the current image information meets the task requirements, it includes at least one of the following judgment criteria: Determine whether the target observation area is within the visible range of the initial observation image set; Determine whether the target object is fully visible in the image; Determine whether the target object is obscured by vehicles, pedestrians, obstacles, or road structures; Determine whether the relative orientation and spatial distance between the target object and the vehicle can be determined; Determine whether the current image lacks information about side, rear, top, or local target regions; Determine whether the scale of the target object in the current image is lower than a preset analyzable scale threshold; If at least one of the judgment criteria does not meet the preset conditions, the current image information is determined not to meet the task requirements.

5. The virtual camera perspective generation method for autonomous driving simulation according to claim 1, characterized in that, In S3, the observation area field is used to represent the spatial range to be observed. The observed object field is used to represent vehicles, pedestrians, obstacles, lane boundaries, or other traffic participants; The viewpoint type field is used to represent side view, rear view, top view, oblique view, or local target view. The orientation category field is used to indicate the position of the observation area relative to the front, rear, left, right, left front, right front, left rear, or right rear of the vehicle. The missing information reason field is used to indicate occlusion, blind spot, too small scale, unclear spatial relationship or incomplete target; The observation priority field is used to indicate the importance of supplementary observations.

6. The virtual camera perspective generation method for autonomous driving simulation according to claim 1, characterized in that, In S4, the viewpoint parameters include relative pose parameters and field of view parameters; The relative pose parameters include lateral offset, longitudinal offset, height offset, azimuth adjustment, pitch adjustment, and roll adjustment. The field of view parameter is used to represent the field of view range of the virtual camera; The relative pose parameters and the field of view parameters are used together to determine the position offset, attitude adjustment and field of view range of the virtual camera relative to the camera pose anchor point.

7. The virtual camera viewpoint generation method for autonomous driving simulation according to claim 6, characterized in that, In S5, the calculation of the target pose of the virtual camera based on the camera pose anchor point and viewpoint parameters specifically includes: The field of view of the virtual camera is obtained based on the field of view angle parameter; Based on the camera pose anchor point and relative pose parameters, relative pose combination or coordinate transformation is performed to obtain the target position and target orientation of the virtual camera in the autonomous driving simulation environment, and spatial constraint checks are performed on the target position and target orientation. The spatial constraint check includes: determining whether the target location is within the renderable area, determining whether the target location has a spatial conflict with vehicles, pedestrians, obstacles or road structures, determining whether the target orientation can cover the target observation area, and determining whether there is severe occlusion between the virtual camera and the observed target; when the preset spatial constraint conditions are not met, the virtual camera target position, height, distance or orientation is corrected.

8. The virtual camera viewpoint generation method for autonomous driving simulation according to claim 7, characterized in that, In S5, the virtual camera is controlled to acquire supplementary viewpoint images and feed them back to the visual language model, forming a closed-loop iteration until the current image information meets the task requirements or reaches the preset maximum number of iterations, and finally outputs the scene understanding result, specifically including: After satisfying the preset spatial constraints, the virtual camera in the autonomous driving simulation environment is controlled to acquire supplementary viewpoint images, and the supplementary viewpoint images are fed back to the visual language model, so that the visual language model can combine the initial observation image set and the supplementary viewpoint images to continue to perform scene understanding and information sufficiency judgment. If the current image information still does not meet the task requirements, repeat steps S3-S5 to generate structured virtual viewpoint requirements, generate viewpoint parameters, calculate virtual camera target pose, acquire supplementary viewpoint images, and provide image feedback, until the visual language model determines that it has obtained image information that meets the task requirements, or reaches the preset maximum number of iterations, and finally outputs the scene understanding result.

9. A virtual camera view generation system for autonomous driving simulation, applied to the virtual camera view generation method for autonomous driving simulation as described in any one of claims 1-8, characterized in that, include: The simulation state acquisition module is used to acquire the vehicle's operating state, scene state information, and multiple initial observation images in the autonomous driving simulation environment. The initial image input module is used to organize the multiple initial observation images according to their corresponding viewpoint, image source, or acquisition time to form an initial observation image set, and input the initial observation image set into a pre-constructed visual language model; The information sufficiency determination module is used to perform joint scene understanding on the initial observation image set based on the visual language model, and to determine whether the current image information meets the task requirements. The viewpoint requirement parsing module is used to generate and parse structured virtual viewpoint requirements when the current image information does not meet the task requirements. The pose anchor point establishment module is used to establish camera pose anchor points based on the vehicle pose or the pose of the vehicle camera. The viewpoint parameter generation module is used to match a preset viewpoint parameter template according to the structured virtual viewpoint requirements and generate viewpoint parameters relative to the camera pose anchor point. The virtual camera pose calculation module is used to calculate the target pose of the virtual camera based on the camera pose anchor points and view parameters; The pose constraint correction module is used to check the spatial constraint of the virtual camera target pose and correct it when the preset spatial constraint conditions are not met. The virtual camera rendering control module is used to control the virtual camera in the autonomous driving simulation environment to acquire supplementary viewpoint images; The image feedback and recording module is used to feed back the supplementary viewpoint image to the visual language model and record the generation information corresponding to the supplementary viewpoint image; The closed-loop iterative management module is used to repeatedly execute the processes of information sufficiency determination, viewpoint requirement analysis, viewpoint parameter generation, virtual camera pose calculation, supplementary viewpoint image acquisition, and image feedback until the visual language model determines that it has obtained image information that meets the task requirements, or reaches the preset maximum number of iterations.

Images