Three-dimensional scene recognition method and system based on visual language large model

Abstract

The embodiment of the application provides a three-dimensional scene recognition method and system based on a visual language large model, and belongs to the technical field of three-dimensional scene recognition; the method comprises the following steps: collecting multi-modal three-dimensional space data of a real three-dimensional scene, extracting scene features and performing semantic recognition to obtain a visual feature dataset; calculating a current conditional semantic entropy value according to a probability correlation table; identifying a scene subtype by analyzing continuous scene flow data in the multi-modal three-dimensional space data; calculating a dynamic conditional semantic entropy threshold value; generating complete prompt words corresponding to the scene features through a rule engine; and performing collaborative reasoning through a multi-modal model to obtain a three-dimensional scene recognition result. The complete prompt words generated based on the rule engine convert three-dimensional visual features into structured semantic instructions, dynamically guide the model to focus on key scene elements, and effectively cope with the recognition requirements of complex dynamic scenes.

Description

Technical Field

[0001] This invention relates to the field of 3D scene recognition technology, specifically to a 3D scene recognition method and system based on a large visual language model. Background Technology

[0002] In the interdisciplinary field of computer vision and artificial intelligence, 3D scene recognition technology has shown rapid development in recent years. In its early stages, 3D scene recognition mainly relied on manually designed feature extraction methods. Traditional algorithms such as scale-invariant feature transform and accelerated robust feature transformation were typical examples of this period. These methods use manually defined feature templates to match key information in the scene, and can play a certain role in recognizing scenes with simple structures and a small number of objects.

[0003] With the rise of deep learning technology, convolutional neural networks have been widely applied in the field of 3D scene recognition. Through the automatic learning mechanism of multi-layer neural networks, intelligent feature extraction has been achieved, significantly improving the accuracy of recognition tasks. For example, network models specifically designed for point cloud data can directly process point sets in 3D space, effectively extracting geometric features such as the 3D contours and spatial relationships of objects. However, these deep learning-based methods still have significant shortcomings. They mostly focus on the extraction and analysis of geometric features, and their ability to understand the semantic relationships between objects in a scene and the semantic attributes of the overall scene remains limited, making it difficult to meet the needs of accurate semantic recognition in complex scenes. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for 3D scene recognition based on a large visual language model, which is used to generate complete prompt words that adapt to scene features based on multimodal data, and to achieve 3D scene recognition with the assistance of prompt words.

[0005] To achieve the above objectives, this invention provides a 3D scene recognition method based on a large visual language model, comprising: collecting multimodal 3D spatial data of real 3D scenes, extracting scene features and performing semantic recognition to obtain a visual feature dataset; constructing a probability association table based on the visual feature dataset, and calculating the current conditional semantic entropy value hierarchically according to the probability association table; identifying scene subtypes by analyzing continuous scene stream data in the multimodal 3D spatial data; matching a baseline conditional semantic entropy threshold according to the identified scene subtypes, and calculating a dynamic conditional semantic entropy threshold by combining the baseline conditional semantic entropy threshold and the entropy standard deviation of the current conditional semantic entropy value; generating complete prompt words corresponding to the scene features through a rule engine based on the current conditional semantic entropy value and the dynamic conditional semantic entropy threshold; and inputting the visual features extracted based on the visual feature dataset and continuous scene stream data, and the complete prompt words, into a multimodal model for collaborative reasoning to obtain a 3D scene recognition result.

[0006] Optionally, the step of collecting multimodal 3D spatial data of real 3D scenes, extracting scene features, and performing semantic recognition to obtain a visual feature dataset includes: extracting scene features from the multimodal 3D spatial data, wherein the scene features include geometric features, material features, and spatial relationships of objects; preprocessing the scene features; performing semantic parsing on the multimodal 3D spatial data using a 3D convolutional neural network model to generate semantic labels; and associating and mapping the geometric feature parameters, material feature parameters, spatial relationship values, and the semantic labels to obtain a visual feature dataset.

[0007] Optionally, the step of constructing a probability association table based on the visual feature dataset and calculating the current conditional semantic entropy value hierarchically according to the probability association table includes: discretizing the geometric feature parameters, material feature parameters, spatial relationship values, and semantic labels in the visual feature dataset; analyzing the co-occurrence frequency of different scene feature combinations and the semantic labels, and constructing a probability association table based on the co-occurrence frequency; hierarchically dividing the semantic labels and scene features according to a preset hierarchical dimension; labeling the conditional variables and target variables of each layer after hierarchical division, and calculating the current conditional semantic entropy value of each layer based on the conditional probabilities in the probability association table.

[0008] Optionally, the step of stratifying the semantic tags and the scene features according to a preset stratification dimension includes: stratifying the semantic tags according to semantic granularity and stratifying the scene features according to combination type.

[0009] Optionally, the step of identifying scene subtypes by analyzing continuous scene stream data in the multimodal three-dimensional spatial data includes: performing temporal alignment, noise filtering, and sliding window segmentation on the continuous scene stream data to obtain sub-segments; encoding the temporal features of each window through a neural network model; and identifying scene subtypes through a pre-trained classification network.

[0010] Optionally, the step of matching a baseline conditional semantic entropy threshold based on the identified scene subtype, and calculating a dynamic conditional semantic entropy threshold by combining the baseline conditional semantic entropy threshold and the entropy standard deviation of the current conditional semantic entropy value, includes: retrieving the baseline conditional semantic entropy threshold corresponding to the scene subtype from a scene subtype baseline entropy value lookup table pre-constructed based on historical scene data; extracting the current conditional semantic entropy value sequence calculated hierarchically within the current window, and calculating the entropy standard deviation of the sequence; and calculating the dynamic conditional semantic entropy threshold based on the entropy standard deviation and the baseline conditional semantic entropy threshold.

[0011] Optionally, the step of generating a complete prompt word corresponding to the scene features through a rule engine based on the current conditional semantic entropy value and the dynamic conditional semantic entropy threshold includes: comparing the current conditional semantic entropy value and the dynamic conditional semantic entropy threshold of each layer, and retrieving matching rules from a preset rule library based on the comparison results; retrieving the corresponding basic prompt word framework from a semantic template library based on the scene subtype identification results; filling the basic prompt word framework according to the retrieved matching rules; extracting scene feature parameters within the current window, converting these parameters into natural language descriptions and embedding them into basic prompt words; and performing logical coherence verification on the basic prompt word combination fragments through a rule engine, and generating a complete prompt word corresponding to the scene features after the verification passes.

[0012] Optionally, the step of inputting the visual features extracted based on the visual feature dataset and continuous scene stream data, and the complete prompt word into a multimodal model for collaborative reasoning to obtain a 3D scene recognition result includes: preprocessing the visual features extracted based on the visual feature dataset and continuous scene stream data; semantically encoding the complete prompt word using a pre-trained language model; inputting the preprocessed visual features and the semantically encoded complete prompt word into the multimodal model, and the multimodal model outputting a 3D scene recognition result.

[0013] Optionally, the step of inputting the visual features extracted from the visual feature dataset and continuous scene stream data, along with the complete prompt words, into a multimodal model for collaborative reasoning to obtain a 3D scene recognition result further includes: the multimodal model fusing the preprocessed visual features and the semantically encoded complete prompt words to obtain a fused feature vector; the multimodal model parsing the fused feature vector based on a preset 3D scene recognition knowledge base, identifying objects in the 3D scene and generating the 3D coordinate information of the objects; reasoning about the overall structure and type of the 3D scene based on the spatial relationships between objects and the semantic description of the complete prompt words; and integrating and evaluating the reasoning results to obtain a 3D scene recognition result.

[0014] On the other hand, the present invention provides a three-dimensional scene recognition system based on a large visual language model, for implementing a three-dimensional scene recognition method based on a large visual language model. The system includes a control module, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the three-dimensional scene recognition method based on a large visual language model.

[0015] The aforementioned technical solution, through deep fusion of multimodal 3D spatial data, achieves comprehensive capture of scene geometry, material, and spatial relationship features. By employing probabilistic association tables and hierarchical conditional semantic entropy, it enhances the hierarchy and refinement of semantic parsing, strengthening the ability to mine semantic associations in complex scenes. The introduction of dynamic conditional semantic entropy thresholds enables the system to adaptively adjust to scene subtypes, improving recognition robustness. Complete prompts generated by the rule engine transform 3D visual features into structured semantic instructions, dynamically guiding the model to focus on key scene elements; improving the accuracy and generalization ability of 3D scene recognition, effectively addressing the recognition needs of complex dynamic scenes.

[0016] Other features and advantages of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the following detailed description to explain the invention, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of 3D scene recognition based on a large visual language model.

[0018] Figure 2 This is a flowchart for calculating the dynamic conditional semantic entropy threshold. Detailed Implementation

[0019] The following is in conjunction with the appendix Figure 1 - Appendix Figure 2The specific implementation methods of the embodiments of the present invention will be described in detail below. It should be understood that the specific implementation methods described herein are only for illustrating and explaining the embodiments of the present invention, and are not intended to limit the embodiments of the present invention.

[0020] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application all comply with the relevant provisions of national laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0021] In the process of realizing this invention, the inventors of this application discovered that the existing technology has the defects that feature extraction is only applicable to simple scenarios, and deep learning focuses on geometric features and ignores semantic associations, making it difficult to meet the recognition needs of complex dynamic scenarios.

[0022] Example 1 Reference Figures 1-2 This is the first embodiment of the present invention, which provides a 3D scene recognition method based on a large visual language model, including: S100: Collect multimodal 3D spatial data of real 3D scenes, extract scene features and perform semantic recognition to obtain a visual feature dataset.

[0023] Furthermore, scene features are extracted from multimodal 3D spatial data, including geometric features, material features, and spatial relationships of objects; the scene features are preprocessed; semantic parsing of the multimodal 3D spatial data is performed using a 3D convolutional neural network model to generate semantic labels; and geometric feature parameters, material feature parameters, spatial relationship values, and semantic labels are correlated and mapped to obtain a visual feature dataset.

[0024] Specifically, precise 3D point cloud data of the scene is acquired through LiDAR scanning, while color images and depth maps are simultaneously acquired using a depth camera and converted into spatial point sets with distance information. Next, scene features are extracted. Geometric features include calculating the aspect ratio of objects, surface curvature, and symmetry index (the degree of overlap of point sets on both sides of a symmetry plane, ranging from 0 to 1). Material features are identified from the color images, including wood grain and metallic reflection, and reflectivity values ​​are recorded. Spatial relationships are calculated, including vertical distance, horizontal spacing, and azimuth angle (0-360°). The extracted raw scene features undergo necessary preprocessing, and then a 3D convolutional neural network model is used to perform semantic parsing on the multimodal 3D spatial data, generating semantic labels such as object categories (e.g., "chandelier," "bookshelf") and region labels (e.g., "desktop," "ceiling"). Finally, geometric parameters, material parameters, spatial relationship values, and semantic labels are mapped and integrated into a comprehensive dataset containing shape, material, spatial relationships, and semantic categories, resulting in a comprehensive visual feature dataset that integrates geometric parameters, material characteristics, spatial relationship values, and semantic labels.

[0025] Preferably, the above solution integrates LiDAR point clouds and depth camera RGB-D images to comprehensively collect multimodal data such as the geometric structure, surface texture of objects, and precise spatial relationships of the 3D scene. It then uses a 3D convolutional neural network for semantic parsing to associate low-level visual features with high-level semantic concepts, thus constructing a structured comprehensive visual feature dataset. This provides a unified, rich, and semantically rich basic data source for subsequent processing.

[0026] S200: Based on the visual feature dataset, construct a probability association table, and calculate the current conditional semantic entropy value hierarchically according to the probability association table.

[0027] Furthermore, the geometric feature parameters, material feature parameters, spatial relationship values, and semantic labels in the visual feature dataset are discretized; the co-occurrence frequency of different scene feature combinations and semantic labels is analyzed, and a probability association table is constructed based on the co-occurrence frequency; the semantic labels and scene features are layered according to the preset layering dimension; the condition variables and target variables of each layer are labeled, and the current conditional semantic entropy value of each layer is calculated based on the conditional probability in the probability association table.

[0028] Furthermore, semantic tags are layered according to semantic granularity, and scene features are layered according to combination type.

[0029] Specifically, the geometric feature parameters, material feature parameters, spatial relationship values, and semantic labels in the visual feature dataset are discretized (for example, continuous values ​​are divided into predefined intervals or categories). Then, the co-occurrence frequency of different scene feature combinations and semantic labels is analyzed, that is, the probability of a specific semantic label (such as the object category "chandelier" and the area label "desktop") appearing under a specific geometric shape, specific material, and specific spatial relationship feature combination is statistically analyzed. Based on the co-occurrence frequency statistics, a hierarchical probability association table is constructed (usually a three-level table: Level 1 is the conditional probability of geometric features → material; Level 2 is the conditional probability of geometric features + material features → spatial relationships; Level 3 is the conditional probability of shape features + material features + spatial relationships → semantic tags). Next, semantic tags are layered according to semantic granularity (e.g., coarse-grained: "furniture", medium-grained: "storage furniture", fine-grained: "bookshelf"). Scene features are layered according to combination type (e.g., Level 1: geometric features only; Level 2: geometric features + material features; Level 3: geometric features + material features + spatial relationships). Conditional variables (e.g., geometric features in Level 1) and target variables (e.g., material features in Level 1) are labeled for each layer. Finally, based on the conditional probabilities in the probability association table, layered calculations are performed following the information entropy chain rule, calculating the current conditional semantic entropy value and base entropy value for each layer (e.g., ...). =The entropy of the geometric feature itself, conditional entropy; =Conditional entropy of material distribution when geometric characteristics are known. =Conditional entropy of spatial relationship distribution when geometric and material characteristics are known), the final current conditional semantic entropy value is the sum of the entropy values ​​of each layer, that is .

[0030] Preferably, the above scheme constructs a hierarchical probability association table to quantify the conditional probability relationship between different combinations of visual features and semantic labels. Based on this, it calculates the current conditional semantic entropy value hierarchically according to the principles of information theory, which can objectively quantify the complexity and recognition uncertainty of the current scene.

[0031] S300: Identifies scene subtypes by analyzing continuous scene stream data in multimodal 3D spatial data.

[0032] Furthermore, the continuous scene stream data is processed by temporal alignment, noise filtering, and sliding window segmentation to obtain sub-segments; the temporal features of each window are encoded by a neural network model, and the scene subtypes are identified by a pre-trained classification network.

[0033] Specifically, time-series alignment of LiDAR point clouds and depth camera RGB-D images is achieved through timestamp synchronization (error < 10ms). A bilateral filter removes outlier noise from the point cloud, and a median filter eliminates impulse noise in the depth map to complete noise filtering. Then, the data stream is segmented at fixed intervals (e.g., 5 seconds) to generate equal-length sliding window sub-segments. A neural network model, such as a combination of 1D-CNN (one-dimensional convolutional neural network) and LSTM (long short-term memory network), encodes the temporal features of each window. Three types of features—geometric dynamics within the window (root mean square error of object displacement), material stability (standard deviation of reflectance intensity of key objects), and spatial density (mean number of objects per unit volume)—are used as inputs. The 1D-CNN model extracts local temporal patterns, and the LSTM unit captures global temporal dependencies, outputting a 128-dimensional temporal feature vector. Finally, the feature vectors are parsed by a pre-trained lightweight classification network (such as a ResNet-18 variant or an 18-layer residual network variant), and the scene subtype labels (warehouse: full warehouse / empty warehouse; office: open office area / independent cubicle; exhibition hall: dense exhibition area / sparse exhibition area) and confidence scores (calculated by the maximum probability value output by the softmax layer of the classification network, i.e., the highest confidence probability of predicting the scene subtype) are output. Only when the confidence score is greater than or equal to the preset confidence threshold is it determined to be a valid classification result (otherwise, data reprocessing is triggered), and the output is the valid scene subtype label and its confidence score.

[0034] Preferably, the above scheme extracts dynamic features within a window through temporal alignment, noise filtering, and sliding window segmentation, and then encodes and classifies them using a temporal neural network model. This enables precise identification of dynamic subtypes of a scene. Combined with a confidence-based filtering mechanism, it provides more specific and reliable contextual information about the current environment, enhancing its relevance and accuracy.

[0035] S400: Match the baseline conditional semantic entropy threshold based on the identified scene subtype, and calculate the dynamic conditional semantic entropy threshold by combining the baseline conditional semantic entropy threshold and the entropy standard deviation of the current conditional semantic entropy value.

[0036] Furthermore, the baseline conditional semantic entropy threshold corresponding to the scene subtype is retrieved from the baseline entropy value lookup table pre-built based on historical scene data; the current conditional semantic entropy value sequence calculated hierarchically within the current window is extracted, and the entropy standard deviation of the sequence is calculated; the dynamic conditional semantic entropy threshold is calculated based on the entropy standard deviation and the baseline conditional semantic entropy threshold.

[0037] Specifically, firstly, the baseline conditional semantic entropy threshold matching the currently identified scene subtype is retrieved from a pre-built baseline entropy value lookup table for scene subtypes (e.g., for warehouses: a full warehouse corresponds to a baseline threshold of 0.2, and an empty warehouse corresponds to a baseline threshold of 0.4). Then, the current conditional semantic entropy value sequence of 10 consecutive layers within the current sliding window is extracted, and the standard deviation of the entropy value of this sequence is calculated. To quantify the degree of fluctuation in scenario complexity, The standard deviation of the entropy values ​​for 10 consecutive sets within the current window is used; then, according to the formula: Dynamic conditional semantic entropy threshold = Baseline conditional semantic entropy threshold + Correction amount, where the correction amount = min(0.1, 0.1 × ... ), to make dynamic adjustments (when When ≥1.0, the correction value is taken as the upper limit of 0.1. When <1.0, the correction value is 0.1× The dynamic conditional semantic entropy threshold is strictly limited to the range of 0.0-1.0. When the calculated dynamic conditional semantic entropy threshold is <0, the dynamic conditional semantic entropy threshold is set to 0; when the calculated dynamic conditional semantic entropy threshold is >1, the dynamic conditional semantic entropy threshold is set to 1, and the calculated value remains unchanged in other cases. The output is a dynamic conditional semantic entropy threshold that adapts to the real-time changes of the current scene, and automatically excludes values ​​due to ±3. Invalid entropy values ​​filtered by the rules.

[0038] Preferably, the above scheme obtains the corresponding benchmark conditional semantic entropy threshold from a preset benchmark entropy value lookup table, and analyzes the volatility of the continuous entropy value sequence within the current window (calculating the standard deviation of the entropy value). Based on this, the threshold is dynamically adjusted. An adaptive dynamic conditional semantic entropy threshold is implemented. This threshold is no longer fixed, but is adjusted according to the inherent characteristics of the scene (subtype benchmark) and real-time changes (the degree of fluctuation reflected by the standard deviation), which can more accurately respond to different scenes and their dynamic changes.

[0039] S500: Based on the current conditional semantic entropy value and the dynamic conditional semantic entropy threshold, the rule engine generates complete prompt words corresponding to scene features.

[0040] Furthermore, the current conditional semantic entropy value and the dynamic conditional semantic entropy threshold of each layer are compared, and matching rules in the preset rule library are retrieved based on the comparison results; the corresponding basic prompt word framework is retrieved from the semantic template library based on the scene subtype recognition results; the basic prompt word framework is filled in according to the retrieved matching rules; scene feature parameters in the current window are extracted, and these parameters are converted into natural language descriptions and embedded into basic prompt words; the logical coherence of the basic prompt word combination fragments is checked by the rule engine, and after the check passes, a complete prompt word corresponding to the scene features is generated.

[0041] Specifically, the current conditional semantic entropy value and the dynamic conditional semantic entropy threshold are compared layer by layer. If the current conditional semantic entropy value of any layer is greater than the dynamic conditional semantic entropy threshold, high-complexity rules in the preset rule base are retrieved and all matching rules are executed, such as "prioritizing the detection of special structures". If the current conditional semantic entropy value of any layer is less than or equal to the dynamic conditional semantic entropy threshold, low-complexity rules in the preset rule base are retrieved and only core spatial relationship rules are retained, such as "simplifying redundant modifiers".

[0042] Furthermore, based on the comparison results, rules matching the current scene features are extracted from a preset rule library. Matching rules include: when the circular outline has a symmetry index ≥ 0.8, add "contains a circular structure with {object name}"; when the symmetry index ≥ 0.9 and the area ≥ 2㎡, add "{region name} has a symmetrical layout", etc. Here, "circular outline with a symmetry index ≥ 0.8" and "symmetry index ≥ 0.9 and area ≥ 2㎡" are both visual feature triggering conditions, while "contains a circular structure with {object name}" and "{region name} has a symmetrical layout" are both guide word generation rules.

[0043] Furthermore, based on the scene subtype recognition results, i.e., based on the scene subtype label (such as "full warehouse" or "open office area"), the corresponding basic framework (such as "shelves {location} storing {objects}") is retrieved from the semantic template library (pre-built based on the domain rules of the scene subtype). Then, the real-time scene feature parameters are parsed and converted into natural language descriptions (such as "the storage area has a symmetrical layout"), and the basic framework is embedded according to the matching rules (such as filling "{area name}" → "storage area"). Finally, the spatial relationship rationality is verified by the rule engine (such as verifying that "boxes are stacked on top of the shelves" conforms to gravity constraints, and rejecting contradictory combinations such as "chandeliers are buried in the ground"), to ensure that the prompt words conform to the real physical logic. The output is the complete prompt words that pass the verification (multi-element descriptions are generated for high-complexity scenes, such as "the storage area has a symmetrical layout and contains circular mechanical parts", and simplified descriptions are output for low-complexity scenes, such as "pallet truck is located on the right side of the shelf").

[0044] Preferably, the above scheme matches rules based on the comparison between the current conditional semantic entropy value and the dynamic conditional semantic entropy threshold, and retrieves the basic framework from the template library in conjunction with the scene subtype. It then converts real-time scene feature parameters into a natural language embedding framework, and finally generates complete prompt words through logical verification by a rule engine. This generates context-aware, physically reasonable, and logically consistent natural language prompts that adapt to scene complexity during recognition, ensuring appropriate detail. The rule engine's physical logic verification (such as gravity constraints) eliminates contradictory descriptions. The generated prompt words accurately extract key visual and semantic information from the scene, providing high-quality, structured text guidance for subsequent multimodal models.

[0045] S600: Visual features and complete prompt words extracted from visual feature datasets and continuous scene stream data are input into a multimodal model for collaborative reasoning to obtain 3D scene recognition results.

[0046] Furthermore, the visual features extracted from the visual feature dataset and continuous scene stream data are preprocessed; the complete prompt words are semantically encoded using a pre-trained language model; the preprocessed visual features and the semantically encoded complete prompt words are input into a multimodal model, and the multimodal model outputs the 3D scene recognition results.

[0047] Furthermore, the multimodal model fuses the preprocessed visual features and the semantically encoded complete prompts to obtain a fused feature vector. Based on a pre-defined 3D scene recognition knowledge base, the multimodal model parses the fused feature vector, identifies objects in the 3D scene, and generates the 3D coordinate information of the objects. Based on the spatial relationships between objects and the semantic description of the complete prompts, the model infers the overall structure and type of the 3D scene. The inference results are integrated and evaluated to obtain the 3D scene recognition result.

[0048] Specifically, the input consists of static features (geometric / material / spatial relationships) extracted from a visual feature dataset, dynamic features (geometric dynamism, material stability, spatial density) extracted from continuous scene stream data, and the generated complete prompt words. The processing first preprocesses the visual features, fusing static and dynamic features, standardizing the geometric parameters using Z-score normalization, and normalizing the material parameters to the [0,1] range. A 3D model is then used. The rules filter out abnormal feature values. Simultaneously, a pre-trained language model, such as a variant of BERT, semantically encodes the complete cue words, generating text vectors to capture spatial relationships and object association logic. The pre-processed visual features and complete cue words are then input into a multimodal model (a collaborative inference engine integrating visual features and cue word semantics), and feature fusion is performed through a text-visual cross-attention mechanism to generate a joint feature vector.

[0049] Preferably, a strategy that adjusts the text-visual attention weights based on scene subtype (such as enhancing text guidance in complex scenes and enhancing visual features in simple scenes) can improve the fusion effect.

[0050] Furthermore, based on joint feature vector analysis, the 3D scene is analyzed, including identifying object categories and generating 3D coordinates (calculated by combining point cloud data and depth map backprojection). The overall structural type of the scene (e.g., "warehouse core area") is inferred based on prompt keywords (e.g., "symmetrical layout"). A comprehensive confidence score is generated by weighted aggregation of visual confidence and textual logical consistency scores. Finally, a structured 3D scene recognition result is output, including a list of objects (with names, 3D coordinates, and confidence scores), a spatial relationship topology map, and scene type labels.

[0051] Preferably, valid data is accumulated by category according to scenario subtype. When the number of valid data reaches 50, only entries in the probability association table with an error (absolute deviation between the conditional probability prediction value and the actual observation frequency) ≥ 0.1 are replaced, driving the dynamic update of the probability association table.

[0052] Preferably, the above scheme utilizes a text-visual cross-attention mechanism for deep feature fusion and collaborative reasoning, which improves the overall recognition accuracy and understanding depth of 3D scenes. Visual features provide detailed information, while prompts provide high-level semantic guidance and relational constraints. The complementary nature of these two features enhances the model's ability to recognize objects, understand spatial relationships, infer scene structure, and determine overall type. The strategy of dynamically adjusting the text-visual attention weights further optimizes the fusion effect. The output is structured (object list + coordinates + relationship graph + scene labels) and includes a comprehensive confidence assessment, providing downstream applications with comprehensive and reliable 3D scene cognition.

[0053] The present invention also provides a 3D scene recognition system based on a large visual language model, for implementing a 3D scene recognition method based on a large visual language model. The system includes a control module, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the 3D scene recognition method based on a large visual language model.

[0054] This invention provides a storage medium storing a program that, when executed by a processor, implements a 3D scene recognition method based on a large visual language model.

[0055] This invention provides a processor for running a program, wherein the program executes a 3D scene recognition method based on a large visual language model.

[0056] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements a 3D scene recognition method based on a large visual language model. The device described herein can be a server, PC, PAD, mobile phone, etc.

[0057] This application also provides a computer program product that, when executed on a data processing device, is suitable for performing a 3D scene recognition method based on a large visual language model.

[0058] Those skilled in the art will understand that embodiments of this application can provide methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0059] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0060] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0061] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0062] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0063] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0064] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0065] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0066] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A 3D scene recognition method based on a large visual language model, characterized in that, include: Collect multimodal 3D spatial data of real 3D scenes, extract scene features and perform semantic recognition to obtain a visual feature dataset; Based on the visual feature dataset, a probability association table is constructed, and the current conditional semantic entropy value is calculated hierarchically according to the probability association table; Scene subtypes are identified by analyzing continuous scene stream data in the multimodal 3D spatial data; Based on the identified scene subtype, a baseline conditional semantic entropy threshold is matched, and the dynamic conditional semantic entropy threshold is calculated by combining the baseline conditional semantic entropy threshold with the entropy standard deviation of the current conditional semantic entropy value. Based on the current conditional semantic entropy value and the dynamic conditional semantic entropy threshold, a complete prompt word corresponding to the scene features is generated by the rule engine; Visual features extracted from visual feature datasets and continuous scene stream data, along with the complete prompt words, are input into a multimodal model for collaborative reasoning to obtain 3D scene recognition results.

2. The 3D scene recognition method based on a large visual language model according to claim 1, characterized in that, The process involves collecting multimodal 3D spatial data from real 3D scenes, extracting scene features, and performing semantic recognition to obtain a visual feature dataset, including: Scene features are extracted from multimodal 3D spatial data, including geometric features, material features, and spatial relationships of objects; The scene features are preprocessed; The multimodal 3D spatial data is semantically parsed using a 3D convolutional neural network model. Generate semantic tags; By associating and mapping geometric feature parameters, material feature parameters, spatial relationship values, and semantic labels, a visual feature dataset is obtained.

3. The 3D scene recognition method based on a large visual language model according to claim 1, characterized in that, The step of constructing a probability association table based on the visual feature dataset and calculating the current conditional semantic entropy value hierarchically according to the probability association table includes: The geometric feature parameters, material feature parameters, spatial relationship values, and semantic labels in the visual feature dataset are discretized. Analyze the co-occurrence frequency of different scene feature combinations and the semantic tags, and construct a probability association table based on the co-occurrence frequency; The semantic tags and scene features are layered according to a preset layering dimension; After labeling the conditional variables and target variables of each layer, calculate the current conditional semantic entropy value of each layer based on the conditional probabilities in the probability association table.

4. The 3D scene recognition method based on a large visual language model according to claim 3, characterized in that, The step of stratifying the semantic tags and scene features according to a preset stratification dimension includes: stratifying the semantic tags according to semantic granularity and stratifying the scene features according to combination type.

5. The 3D scene recognition method based on a large visual language model according to claim 1, characterized in that, The step of identifying scene subtypes by analyzing continuous scene stream data in the multimodal three-dimensional spatial data includes: The continuous scene stream data is processed by time alignment, noise filtering and sliding window segmentation to obtain sub-segments; The temporal features of each window are encoded using a neural network model, and scene subtypes are identified using a pre-trained classification network.

6. The 3D scene recognition method based on a large visual language model according to claim 1, characterized in that, The step of matching a baseline conditional semantic entropy threshold based on the identified scene subtype, and calculating a dynamic conditional semantic entropy threshold by combining the baseline conditional semantic entropy threshold and the entropy standard deviation of the current conditional semantic entropy value, includes: Retrieve the baseline conditional semantic entropy threshold corresponding to the scene subtype from the baseline entropy value lookup table pre-built based on historical scene data; Extract the sequence of current conditional semantic entropy values ​​calculated hierarchically within the current window, and calculate the standard deviation of the entropy values ​​of the sequence; The dynamic conditional semantic entropy threshold is calculated based on the standard deviation of the entropy value and the baseline conditional semantic entropy threshold.

7. The 3D scene recognition method based on a large visual language model according to claim 6, characterized in that, The step of generating complete prompt words corresponding to the scene features through a rule engine based on the current conditional semantic entropy value and the dynamic conditional semantic entropy threshold includes: Compare the current conditional semantic entropy value and the dynamic conditional semantic entropy threshold of each layer, and retrieve the matching rules from the preset rule base based on the comparison results; Based on the scene subtype recognition results, retrieve the corresponding basic prompt word framework from the semantic template library; The basic prompt word framework is filled in according to the retrieved matching rules; Extract scene feature parameters within the current window, convert these parameters into natural language descriptions, and embed basic prompt words; The basic prompt word combination fragments are logically coherently verified by the rule engine. Once the verification is passed, a complete prompt word corresponding to the scene features is generated.

8. The 3D scene recognition method based on a large visual language model according to claim 1, characterized in that, The process involves inputting visual features extracted from visual feature datasets and continuous scene stream data, along with the complete prompt words, into a multimodal model for collaborative reasoning to obtain 3D scene recognition results, including: Preprocess the visual features extracted from the visual feature dataset and continuous scene stream data; Semantic encoding of complete prompt words is performed using a pre-trained language model; The preprocessed visual features and the semantically encoded complete prompt words are input into a multimodal model, which outputs a 3D scene recognition result.

9. The 3D scene recognition method based on a large visual language model according to claim 1, characterized in that, The step of inputting visual features extracted from visual feature datasets and continuous scene stream data, along with the complete prompt words, into a multimodal model for collaborative reasoning to obtain 3D scene recognition results also includes: The multimodal model fuses the preprocessed visual features and the semantically encoded complete prompt words to obtain a fused feature vector; The multimodal model, based on a pre-defined 3D scene recognition knowledge base, parses the fused feature vector, identifies objects in the 3D scene, and generates the 3D coordinate information of the objects. Infer the overall structure and type of the 3D scene based on the spatial relationships between objects and the semantic description of complete prompts; The inference results are integrated and evaluated to obtain the 3D scene recognition results.

10. A 3D scene recognition system based on a large visual language model, characterized in that, The system includes a control module, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the three-dimensional scene recognition method based on a large visual language model according to any one of claims 1-9.

Images