Complex test scene perception and partition construction method based on multi-modal large model

Abstract

The invention discloses a complex test scene perception and partition construction method based on a multi-modal large model, and belongs to the technical field of digital test verification of high-end equipment. According to the method, a scene description prompt is designed, a multi-modal large model is utilized to generate a 2D test scene image with an equidistant view angle, and images with consistent semantics are screened through similarity calculation. Secondly, extracting spatial positions, semantic information and three-dimensional geometric parameters of objects in the scene by adopting a detection segmentation model and a depth estimation technology, and generating a feature map fusing space and semantics; and finally, geometric and texture description of an object is generated based on a multi-modal large model, a 3D grid is constructed by using a 3D generation model, model parameters are adjusted through dense matching and iterative optimization, and category accuracy and spatial coherence are ensured. According to the method, the full-process automation from 2D scene generation to 3D refined reconstruction is realized, and the precision and efficiency of complex test scene construction are remarkably improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of digital testing and verification of high-end equipment, and particularly relates to a complex testing scene perception and partition construction method based on a multi-modal large model. BACKGROUND

[0002] In the field of digital testing of high-end equipment, high-precision modeling of complex scenes is a key prerequisite for verifying the reliability of algorithms and the robustness of systems. Traditional testing scene construction methods mainly rely on manual design or simple image synthesis techniques, which have significant limitations. First, the scene generation is not automated enough, and 2D scenes need to be manually drawn or spliced. It is difficult to quickly generate images that meet the equidistant perspective and semantic consistency according to diversified testing requirements, resulting in low efficiency and high cost. Second, the existing methods have limited analysis capabilities for scene elements, lacking the ability to segment pixels, estimate depth information, and extract semantic features of objects in 2D images, resulting in insufficient precision of key parameters such as spatial position and physical size required for three-dimensional reconstruction. In addition, traditional techniques do not form a complete closed-loop link of "generation-perception-reconstruction-evaluation", making it difficult to ensure that the generated three-dimensional scenes meet the complex testing requirements in terms of category accuracy and spatial coherence.

[0003] With the rapid development of multi-modal large models and computer vision technology, new possibilities are provided to solve the above problems. Multi-modal large models can generate high-quality 2D scene images based on natural language descriptions, while advanced detection and segmentation and depth estimation techniques can accurately extract spatial and semantic information of scene elements. However, how to organically integrate these technologies to achieve full-process automation from 2D scene generation to 3D fine reconstruction is still a technical difficulty that needs to be broken through. SUMMARY

[0004] To solve the above technical problems, the application provides a complex testing scene perception and partition construction method based on a multi-modal large model, which realizes efficient construction and fine partitioning of testing scenes through the cooperation of three modules: "2D scene generation-element perception extraction-3D reconstruction optimization".

[0005] To achieve the above purpose, the application adopts the following technical solutions:

[0006] A complex testing scene perception and partition construction method based on a multi-modal large model, the method comprising:

[0007] Step 1, constructing a structured scene description prompt based on the user's testing requirements, inputting a multi-modal large model to generate effective equidistant perspective 2D testing scene images;

[0008] Step 2, identifying objects in the effective 2D test scene image and obtaining pixel-level segmentation results, generating a scene depth map of the effective 2D test scene image and a high-dimensional image feature map, fusing the scene depth map and the high-dimensional image feature map to obtain a spatial perception feature map, and calculating the three-dimensional position and size of the object based on the spatial perception feature map;

[0009] Step 3, generating natural language descriptions using a multi-modal large model and constructing a 3D model, combining the calculated three-dimensional position and size of the object, adjusting the 3D model parameters through dense matching and iterative optimization algorithms, and completing scene quality assessment with category accuracy and spatial continuity as indicators.

[0010] Further, the step 1 includes:

[0011] Step 1.1, constructing a structured scene description prompt according to the user's test requirements;

[0012] Step 1.2, inputting the structured scene description prompt into a multi-modal large model to generate several equidistant perspective 2D test scene images;

[0013] Step 1.3, filtering the effective 2D test scene images that meet the requirements through semantic similarity calculation.

[0014] Further, the step 2 includes:

[0015] Step 2.1, using a detection and segmentation model to detect and segment objects at the pixel level for the input effective 2D test scene image;

[0016] Step 2.2, using a depth estimation model to process the input effective 2D test scene image to estimate the depth map corresponding to the image and the intrinsic and extrinsic parameters of the camera;

[0017] Step 2.3, combining a self-supervised visual feature extraction model to extract features from the effective 2D test scene image to obtain a high-dimensional image feature map;

[0018] Step 2.4, converting the depth map to the same size as the high-dimensional image feature map, forming a fusion feature map through channel splicing, and obtaining a spatial perception feature map through convolution dimension reduction;

[0019] Step 2.5, based on the spatial perception feature map, accurately analyzing and obtaining the three-dimensional spatial position information and physical size information of different categories of objects in the scene.

[0020] Further, the step 3 includes:

[0021] Step 3.1, using a multi-modal large model to generate natural language descriptions of different categories of objects segmented in the 2D test scene and constructing a 3D grid generation model;

[0022] Step 3.2, intelligently match N pairs of corresponding points between the rendered three-dimensional object surface and the visible area of the object in the original 2D test scene image by a dense matching method; optimize the scale parameter, translation vector, and rotation angle around the vertical axis of each three-dimensional object by iteratively optimizing the algorithm to minimize the L2 loss function of the corresponding positions of the feature point pairs in the 2D image space and the 3D model space, and back-propagating the gradient;

[0023] Step 3.3, evaluate the constructed 3D mesh generation model by setting the category accuracy and spatial continuity of the generated object as evaluation indicators, if any evaluation indicator is lower than the pre-set qualified threshold, return to step 3.1 or step 3.2 to re-execute until all indicators meet the requirements.

[0024] Further, the step 1.1 includes parsing the user input test requirements, and structuring or natural language describing the key objects in the test scene, the interaction between objects, the environmental conditions, and the specific test events.

[0025] Further, the step 2.5 includes decoding the spatial perception feature map using a specially designed prediction head network, the prediction head network is composed of multiple parallel convolution layer branches, each branch is responsible for predicting the position information and physical size information of objects of different categories in three-dimensional space, including the center point coordinates, scale parameters, and rotation angles of the three-dimensional bounding box.

[0026] Further, the step 3.1 includes using a multi-modal large model to generate detailed geometric shape and appearance texture natural language descriptions for each object segmented in step 2, providing semantic guidance for 3D generation; input the segmented object image block and its corresponding natural language description together into the 3D mesh generation model for triangular mesh generation of each object.

[0027] Further, the step 1.3 includes using a text-image contrast model to convert the prompt and image description into a vector to evaluate the cosine similarity between the image and the text.

[0028] In a second aspect, the present application provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned complex test scene perception and partition construction method based on a multi-modal large model.

[0029] Thirdly, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enable the processor to implement the aforementioned method for perceiving and partitioning complex test scenarios based on a multimodal large model.

[0030] The beneficial effects of this invention are as follows:

[0031] Enhance automation: Automatically generate 2D scene images that meet testing requirements through multimodal large models, and filter valid results based on semantic similarity, significantly reducing manual intervention and improving scene construction efficiency.

[0032] Enhanced resolution accuracy: By combining detection and segmentation models with depth estimation technology, pixel-level segmentation, 3D position calibration, and physical size estimation of scene elements are achieved, providing high-precision data support for subsequent 3D reconstruction.

[0033] Optimize 3D reconstruction quality: Utilize multimodal large models to generate geometric and texture descriptions of objects, and combine dense matching and iterative optimization algorithms to ensure the accuracy of 3D model categories and the rationality of spatial layout, meeting the high standards required for complex testing scenarios.

[0034] Forming a complete closed-loop chain: Through the full-process design of "generation-perception-reconstruction-evaluation", the intelligent and iterative optimization of scenario construction is realized, significantly improving the realism and usability of test scenarios. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of a method for perceiving and partitioning complex test scenarios based on a multimodal large model, according to the present invention. Detailed Implementation

[0036] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0037] like Figure 1 As shown, this invention provides a method for complex test scene perception and partitioning construction based on a multimodal large model. The entire method achieves efficient construction and refined partitioning of test scenes through modular design, including three core modules: a 2D test scene generation module, a test scene perception module, and a test scene partitioning and 3D reconstruction optimization module. Specifically, the method includes:

[0038] Step 1: Construct a structured scene description prompt based on test requirements, and input a multimodal large model to generate effective isometric 2D test scene images;

[0039] The above steps are based on the 2D test scene generation module. The goal of this stage is to construct a series of 2D images for generating subsequent complex scene models. Specifically, this includes:

[0040] Step 1.1. Construct a structured scene description prompt according to the user's test requirements;

[0041] Parse the user's input test requirements through the natural language processing module, extract the test object type (such as an autonomous vehicle, urban road obstacles, road facilities), test content (such as intersection obstacle avoidance testing, collision testing, path planning), and perspective requirements (such as "isometric overhead perspective"), or include structured or natural language descriptions of key objects in the test scene, object interaction, environmental conditions, and specific test events.

[0042] Combine the test object characteristics library (predefined object geometric proportions, typical scene layout rules) to generate structured prompt text containing object categories, spatial positions, size proportions, perspective parameters, and lighting conditions, such as: "Urban intersection scene with an isometric perspective, containing 1 red sedan (4.5m long, 1.8m wide), 2 pedestrians (1.7m tall), the sedan is located 3m to the left of the intersection center, the pedestrians are on the right sidewalk, and the light direction is 45° from the upper left."

[0043] Step 1.2. Input the structured scene description prompt into a multi-modal large model to generate an isometric perspective 2D test scene image;

[0044] Use a pre-trained multi-modal generation model (such as the joint architecture of Stable Diffusion and CLIP visual language model) to convert the scene description prompt into a model input tensor.

[0045] By adding perspective control conditions (such as embedding an isometric perspective projection matrix parameter in the model input), force the model to output a 2D test scene image that meets the perspective requirements, with a resolution of 1024 1024, color mode is RGB three channels, to ensure the standardization of scene geometric relationship, reduce perspective distortion, more accurately reflect the relative size and position of objects, facilitate subsequent perception and three-dimensional reconstruction.

[0046] Step 1.3. Select the effective 2D test scene image set that meets the requirements through semantic similarity calculation;

[0047] Perform consistency testing on the generated images, input each generated scene image into the image description branch of the multi-modal large model (such as the BLIP-2 model) to generate image semantic description text.

[0048] Use the CLIP text encoder to vectorize the structured scene description prompt and the generated image semantic description text, respectively, and calculate the cosine similarity between them.

[0049] Set the similarity threshold to 0.85, retain the 2D test scene images with cosine similarity higher than the threshold, eliminate semantic deviation samples, and form an effective 2D test scene image set.

[0050] Step 2, identify objects in the effective 2D test scene image and obtain pixel-level segmentation results, generate scene depth maps and high-dimensional image feature maps of the effective 2D test scene image, fuse the scene depth maps and high-dimensional image feature maps to obtain spatial perception feature maps, and calculate the three-dimensional position and size of the object based on the spatial perception feature maps;

[0051] The above steps are based on the test scene perception module. Specifically:

[0052] Step 2.1, use Mask R-CNN detection and segmentation model (or SegmentAnything) to detect and segment objects in the input effective 2D test scene image.

[0053] Output the class label of each object (such as "car", "pedestrian", "traffic light"), pixel-level mask matrix (HxW, value 0 or 1), and bounding box coordinates (x1, y1, x2, y2).

[0054] Step 2.2, use a depth estimation model to process each effective 2D test scene image to estimate the corresponding depth map, point cloud map, and camera intrinsic and extrinsic information.

[0055] Use monocular depth estimation model (MASt3R, Vggt) to input RGB image to generate dense depth map (HxWx1, depth value unit: meter).

[0056] Through the camera pose estimation module, use the known size objects in the scene (such as road markings drawn in real proportion) to calculate the camera intrinsic matrix K (focal length , , optical center , ) and extrinsic matrix [R|T] (rotation matrix R and translation vector T).

[0057] Step 2.3, combine the self-supervised visual feature extraction model to extract features from the effective 2D test scene image to obtain high-dimensional image feature maps.

[0058] Use self-supervised visual feature extraction model DINO (ViT-Small architecture, input image block size 16x16) to extract high-dimensional semantic feature maps (H / 16xW / 16x384).

[0059] Step 2.4. Fuse the scene depth map with the high-dimensional image feature map to obtain a spatial perception feature map; resize the dense depth map to the same size as the high-dimensional semantic feature map (H / 16 x W / 16 x 1), and form a fusion feature map of H / 16 x W / 16 x 385 by channel splicing.

[0060] After 3-layer convolution dimension reduction (each layer of convolution kernel 3 x 3, channel number 256, 128, 64 in turn, followed by a ReLU activation function), a spatial perception feature map (H / 16 x W / 16 x 64) is obtained, which contains visual semantic and spatial depth information of the object.

[0061] Step 2.5. Based on the spatial perception feature map, accurately analyze and obtain the three-dimensional spatial position information and physical size information of different categories of objects in the scene; for the pixel mask area of each segmented object, extract the feature vector of the corresponding area in the spatial perception feature map, and calculate the three-dimensional spatial coordinates (X, Y, Z) through the coordinate transformation formula:

[0062] ;

[0063] According to the pixel range and depth distribution of the object mask, the minimum bounding box size is calculated to obtain the physical size (length, width, height) of the object, with an accuracy error of within ±5%.

[0064] Step 3. Use a multi-modal large model to generate a natural language description and construct a 3D model, combine the calculated three-dimensional position and size of the object, and adjust the 3D model parameters through dense matching and iterative optimization algorithm to complete scene quality evaluation with category accuracy and spatial continuity as indicators; the above steps are based on the test scene partition and 3D reconstruction optimization module.

[0065] Step 3.1. Use a multi-modal large model to generate a natural language description of the object and construct a 3D mesh generation model;

[0066] Use a multi-modal large model to generate detailed geometric shapes (combination of primitive cubes, cylinders, etc.) and appearance textures (color, material, mapping coordinates) for each object segmented in step 2 (excluding background and plane elements unrelated to the viewing angle); input the segmented object image block (resolution 224 x 224) and its corresponding natural language description into the 3D mesh generation model (such as DreamFusion generation model or Point-E), and perform 3D rendering through the NeRF (Neural Radiance Field) architecture. Align the semantic features of the text description and the rendered image during training to generate a triangular mesh model (vertex number ≥10,000, face number ≥20,000) for each object, supporting subdivision surface optimization to improve surface details.

[0067] Further, the 3D mesh generation model can be an advanced three-dimensional content generation technology based on NeRF (Neural Radiance Field), 3D Gaussian Splatting, text-to-3D diffusion model (such as Shap-E, Point-E, DreamFusion).

[0068] Step 3.2, adjusting the 3D mesh generation model parameters by a dense matching and iterative optimization algorithm; the iterative optimization algorithm aims to minimize the L2 loss function of the corresponding positions of the feature point pairs in the 2D image space and the 3D model space, and the gradient is back-propagated to accurately optimize the scale parameter, the translation vector, and the rotation angle around the vertical axis of each three-dimensional object.

[0069] Orthogonal projection rendering is performed on each 3D model to generate a rendering image with the same perspective as the effective 2D test image, and the position and size information of different categories of objects obtained in step 2.5 is combined to extract the key points in the effective 2D test image and the 3D model rendering image using a SIFT feature detector. A dense corresponding matching method is used to match the feature points by a FLANN matcher.

[0070] The RANSAC algorithm is used to remove the mismatched points and retain the corresponding relationship of at least 50 pairs of valid corresponding points (2D coordinates (x, y) and 3D model coordinates (x, y, z)). , ) and (x, y, z). , , ).

[0071] An optimization objective function is constructed:

[0072] ;

[0073] where s is a scaling factor, is a rotation matrix around the Z axis, T is a translation vector, π is a camera projection function, and K represents the intrinsic parameters of the camera. The L2 loss of the corresponding point pairs is minimized to optimize the scale parameter (scaling factor s), translation vector (T), and rotation angle (θ) around the vertical axis of the three-dimensional model by gradient descent, ensuring that the model pose is consistent with the scene perspective. Preliminary adjustments of the material parameters and lighting parameters can also be included to enhance the visual realism.

[0074] The AdamW optimizer (learning rate 1e-3) is used to iteratively optimize the parameters (s, θ, T) until the L2 loss converges.

[0075] Step 3.3, evaluate the constructed 3D mesh generation model by setting the class accuracy of the generated object and the spatial continuity as evaluation indexes, if any evaluation index is lower than the preset qualified threshold, return to step 3.1 or 3.2 to re-execute until all indexes meet the requirements.

[0076] Set double evaluation indexes - (1) class accuracy, compare the class label generated by the 3D mesh model with the label output by the segmentation module, and require consistency >= 95%, otherwise return to step 3.1 to generate a 3D mesh model again; (2) spatial continuity, calculate the Euclidean distance of the three-dimensional positions of adjacent objects and the geometric mapping error of the pixel distance in the 2D image, and require error < 5% true size, otherwise return to parameter optimization step 3.2.

[0077] The test scene partition result finally output by the method is a three-dimensional scene parameter described in a structured data form (such as a JSON file), which includes the position, pose, size, and class of each object; the three-dimensional model of each object in the scene described in the stl file format can be directly used for subsequent simulation testing or virtual reality applications.

[0078] The above embodiments show that the present application realizes the automatic and high-precision construction of complex test scenes through the deep integration of multi-modal large models and computer vision technology, and has significant engineering application value.

[0079] In a second aspect, the present application provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method for complex test scene perception and partition construction based on multi-modal large models.

[0080] In a third aspect, the present application provides a computer-readable storage medium having stored executable instructions, which when executed by a processor, can enable the processor to implement the aforementioned method for complex test scene perception and partition construction based on multi-modal large models.

[0081] The above-described specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present application, and it should be understood that the above-described specific embodiments are only specific embodiments of the present application and are not intended to limit the present application, and any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A multi-modal large model-based complex test scene perception and partitioning construction method, characterized in that, The method comprises: Step 1, constructing a structured scene description prompt based on the user's test requirements, inputting a multi-modal large model to generate effective equidistant perspective 2D test scene images; Step 2, identifying objects in the effective 2D test scene image and obtaining pixel-level segmentation results, generating a scene depth map and a high-dimensional image feature map of the effective 2D test scene image, fusing the scene depth map and the high-dimensional image feature map to obtain a spatial perception feature map, and calculating the three-dimensional position and size of the object based on the spatial perception feature map; Step 3, generating a natural language description using a multi-modal large model and constructing a 3D model, combining the calculated three-dimensional position and size of the object, adjusting the 3D model parameters through a dense matching and iterative optimization algorithm, and completing scene quality evaluation with category accuracy and spatial continuity as indicators.

2. The method according to claim 1, wherein, The step 1 comprises: Step 1.1, constructing a structured scene description prompt according to the user's test requirements; Step 1.2, inputting the structured scene description prompt into a multi-modal large model to generate several equidistant perspective 2D test scene images; Step 1.3, screening effective 2D test scene images that meet the requirements through semantic similarity calculation.

3. The method of claim 1, wherein, The step 2 comprises: Step 2.1, using a detection and segmentation model to detect and segment objects in the input effective 2D test scene image; Step 2.2, using a depth estimation model to process the input effective 2D test scene image to estimate the depth map corresponding to the image and the intrinsic and extrinsic parameters of the camera; Step 2.3, combining a self-supervised visual feature extraction model to extract features from the effective 2D test scene image to obtain a high-dimensional image feature map; Step 2.4, converting the depth map to the same size as the high-dimensional image feature map, forming a fusion feature map through channel splicing, and obtaining a spatial perception feature map through convolution dimension reduction; Step 2.5, based on the spatial perception feature map, accurately analyzing and obtaining the three-dimensional spatial position information and physical size information of different categories of objects in the scene.

4. The method of claim 1, wherein the method is characterized by, The step 3 comprises: Step 3.1, using a multi-modal large model to generate a natural language description of different categories of objects segmented in the 2D test scene and constructing a 3D grid generation model; Step 3.2, through a dense matching method, intelligently matching N accurate corresponding point pairs between the rendered three-dimensional object surface and the partially visible area of the object in the original 2D test scene image; through an iterative optimization algorithm, taking the minimization of the L2 loss function of the corresponding positions of the feature point pairs in the 2D image space and the 3D model space as the target, propagating the gradient in reverse, and optimizing the scale parameter, translation vector, and rotation angle around the vertical axis of each three-dimensional object; Step 3.3, setting the category accuracy and spatial continuity of the generated object as evaluation indicators to evaluate the constructed 3D grid model, if any evaluation indicator is lower than the pre-set qualified threshold, return to step 3.1 or step 3.2 to execute again until all indicators meet the requirements.

5. The method of claim 2, wherein the method is characterized by, The step 1.1 includes parsing the test requirements input by the user, and structuring or describing the key objects in the test scenario, the interaction relationship between the objects, the environmental conditions, and the specific test events in natural language.

6. The method of claim 3, wherein the method is characterized by, The step 2.5 includes decoding the spatial perception feature map using a specially designed prediction head network, which is composed of multiple parallel convolution layer branches, each branch being responsible for predicting the position information and physical size information of objects of different categories in three-dimensional space, including the center point coordinates, scale parameters, and rotation angles of the three-dimensional bounding box.

7. The method of claim 4, wherein the method is characterized by, The step 3.1 includes using a multi-modal large model to generate detailed geometric shape and appearance texture natural language descriptions for each object segmented in step 2, providing semantic guidance for 3D generation; inputting the segmented object image blocks and their corresponding natural language descriptions together into a 3D mesh generation model for triangular mesh generation of each object.

8. The method of claim 2, wherein the method is characterized by, The step 1.3 includes a semantic similarity calculation method that uses a text-image contrast model to convert prompts and image descriptions into vectors, and evaluates the cosine similarity between the image and the text.

9. An electronic device, comprising: Comprise: One or more processors; Memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the complex test scenario perception and partition construction method based on the multi-modal large model according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, Executable instructions are stored thereon, which when executed by a processor, can enable the processor to implement the complex test scenario perception and partition construction method based on the multi-modal large model according to any one of claims 1-8.

Images