Spatial intelligent multi-mode scene memory method and device and medium

Abstract

The invention provides a spatial intelligence multi-modal scene memory method, and relates to the technical field of spatial intelligence, and the method comprises the steps: constructing a three-dimensional scene structure in a geometric space based on multi-modal data inputted by a sensor, the three-dimensional scene structure being a discretized spatial knowledge graph; establishing a bidirectional constraint relationship between a geometric continuous space and a symbol discrete space for the three-dimensional scene structure; based on the bidirectional constraint relationship, establishing a bidirectional mapping table of a geometric continuous space and a symbol discrete space for each space entity; in addition, the bidirectional mapping table is kept to realize incremental updating; and identifying a bidirectional constraint relationship in the increment by utilizing clustering, and executing local reconstruction on the geometric continuous space and the symbol discrete space. The invention further discloses a spatial intelligent multi-modal scene memory device and a computer storage medium in an exemplary manner.

Description

Technical Field

[0001] This invention relates to the field of spatial intelligence technology; in particular, it relates to a spatial intelligence multimodal scene memory method, device, and medium. Background Technology

[0002] Spatial intelligence systems can be widely applied in intelligent scenarios requiring cross-domain spatial semantic unification and topological adaptive compensation, such as intelligent transportation navigation, seamless indoor and outdoor positioning, augmented reality (AR) spatial recognition, autonomous driving cross-scene navigation, intelligent building management, and multi-source spatial data fusion analysis. However, existing technologies still have significant shortcomings in the collaborative modeling and consistency maintenance between continuous geometry and discrete semantics, making it difficult to support spatial understanding and long-term memory in complex environments. The shortcomings of existing technologies are mainly reflected in the following aspects: First, the spatial representation is singular: Existing spatial intelligence systems typically use a single representation method, either based on continuous geometric space (such as point clouds, voxels) or discrete symbolic space (such as relational graphs). The lack of an effective collaborative mechanism between the two makes it difficult to unify the expression of geometric and semantic information. Second, cross-spatial consistency maintenance is insufficient: changes in geometric space (such as structural changes, new obstacles) often cannot be transmitted to the symbolic space in real time, and the semantic reasoning results in the symbolic space also lack geometric verification, causing spatial cognitive discontinuities and semantic inconsistencies. Third, lack of bidirectional retrieval and reasoning capabilities: Existing systems mostly use unidirectional mapping, unable to flexibly switch reasoning methods between geometric and symbolic spaces, making it difficult to meet the needs of efficient spatial querying and verification in different task scenarios. Fourth, inconsistent memory updates: When new perceptual information is added, most methods only update part of the spatial representation, lacking a synchronous update mechanism between continuous and discrete spaces, which easily leads to spatial drift and semantic bias after long-term operation. Fifth, lack of a geometric-semantic joint reasoning mechanism: Current spatial intelligence models mostly implement geometric reasoning and symbolic reasoning separately, failing to guarantee the consistency of reasoning results in logic and geometry, thus affecting the robustness and interpretability of the system.

[0003] In short, current technologies lack a unified modeling scheme capable of simultaneously handling continuous geometric spaces and discrete symbolic spaces, and cannot achieve cross-space consistency constraints and bidirectional searchable memory update mechanisms. Therefore, there is an urgent need for a new spatial intelligent multimodal scene memory system that integrates continuous and discrete dual spaces to achieve collaborative representation of spatial knowledge, cross-space verification, and long-term consistency maintenance. Summary of the Invention

[0004] This invention provides a spatial intelligent multimodal scene memory method, device and medium, which can solve the problem that the existing technology cannot simultaneously handle the unified modeling of continuous geometric space and discrete symbol space, as well as the problem that it cannot realize cross-space consistency constraints and bidirectional searchable update mechanism.

[0005] Firstly, a spatial intelligent multimodal scene memory method is provided, comprising: constructing a three-dimensional scene structure in geometric space based on multimodal data input from sensors, wherein the three-dimensional scene structure is a discretized spatial knowledge graph; establishing a bidirectional constraint relationship between a geometrically continuous space and a symbolically discrete space for the three-dimensional scene structure; establishing a bidirectional mapping table between the geometrically continuous space and the symbolically discrete space for each spatial entity based on the bidirectional constraint relationship, so that each spatial entity can be accessed through symbolic labels or geometric coordinates recorded in the bidirectional mapping table; monitoring changes in the geometrically continuous space and the symbolically discrete space based on the bidirectional mapping table, and maintaining incremental updates of the bidirectional mapping table; and performing local reconstruction of the geometrically continuous space and the symbolically discrete space by using clustering to identify the bidirectional constraint relationship in the increment.

[0006] In some embodiments, a three-dimensional scene structure is constructed in geometric space based on multimodal data input from sensors, wherein the three-dimensional scene structure is a discretized spatial knowledge graph, including the following steps: constructing a three-dimensional scene structure in geometric space based on multimodal data input from sensors, wherein the multimodal data input from sensors includes: point cloud, depth map, voxel model, RGB image and semantic labels; defining the direction, relationship, function and logical connection between spatial objects based on semantic nodes, relation edges and constraint rules, and constructing a discretized spatial knowledge graph for the three-dimensional scene structure.

[0007] In some embodiments, establishing a bidirectional constraint relationship between a geometrically continuous space and a symbolic discrete space for the three-dimensional scene structure includes the following steps: deriving the adjacency, reachability, or inclusion relationships between spatial objects based on three-dimensional point clouds and topological information, and verifying whether they are consistent with predefined relationships; when inconsistencies are found, correcting the relationships between spatial objects so that the geometric distance, boundary overlap rate, and spatial reachability are consistent with reality.

[0008] In some embodiments, based on the bidirectional mapping table, monitoring changes in the geometrically continuous space and the symbolically discrete space, and maintaining the bidirectional mapping table to achieve incremental updates, includes the following steps: incrementally updating the bidirectional mapping table based on the received multimodal data; and verifying the corresponding bidirectional constraint relationships in the geometrically continuous space and the symbolically discrete space based on the incremental update.

[0009] In some embodiments, the system further includes: verifying the geometrically continuous space and the symbolically discrete space based on a predetermined system period, and triggering a corresponding local reconstruction mechanism when the bidirectional constraint relationship is inconsistent.

[0010] In some embodiments, the system further includes: employing a semantic alignment regularization strategy to correct symbol labels or geometric coordinates in the geometrically continuous space and the symbol discrete space to achieve local semantic consistency.

[0011] Secondly, a spatial intelligent multimodal scene memory device is provided, comprising: a spatial representation module for constructing a three-dimensional scene structure in geometric space based on multimodal data input from sensors, wherein the three-dimensional scene structure is a discretized spatial knowledge graph; a spatial consistency constraint module for establishing a bidirectional constraint relationship between a geometrically continuous space and a symbolically discrete space for the three-dimensional scene structure; a bidirectional searchable space module for establishing a bidirectional mapping table between the geometrically continuous space and the symbolically discrete space for each spatial entity based on the bidirectional constraint relationship, so that each spatial entity can be accessed through symbolic labels or geometric coordinates recorded in the bidirectional mapping table; a spatial joint iterative update module for monitoring changes in the geometrically continuous space and the symbolically discrete space based on the bidirectional mapping table, and maintaining incremental updates of the bidirectional mapping table; and a spatial self-learning module for using clustering to identify the bidirectional constraint relationship in the increment and performing local reconstruction of the geometrically continuous space and the symbolically discrete space.

[0012] In some embodiments, the system further includes: a spatial optimization module, used to verify the geometrically continuous space and the symbolic discrete space based on a predetermined system period, and to trigger a corresponding local reconstruction mechanism when the bidirectional constraint relationship is inconsistent; and a spatial semantic consistency enhancement module, used by the system to adopt a semantic alignment regularization strategy to correct symbol labels or geometric coordinates in the geometrically continuous space and the symbolic discrete space to achieve local semantic consistency.

[0013] Thirdly, an electronic device is provided, comprising: at least one processor; and at least one memory including computer program code, wherein the at least one memory and the computer program code are configured to utilize the at least one processor to cause the electronic device to perform the above-described spatial intelligent multimodal scene memory method.

[0014] Fourthly, a computer-readable storage medium is also provided, wherein program code is stored in the computer-readable storage medium, and the program code can be called by a processor to execute the above-described spatial intelligent multimodal scene memory method.

[0015] Compared with the prior art, the present invention can achieve at least the following beneficial effects: it can simultaneously handle the unified modeling of continuous geometric space and discrete symbolic space, and realize cross-space consistency constraints and bidirectional searchable update mechanism.

[0016] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify essential or necessary features of this disclosure, nor is it intended to limit the scope of this disclosure. Attached Figure Description

[0017] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0018] Figure 1 A schematic diagram of the spatial intelligent multimodal scene memory method provided in an embodiment of this application is shown; Figure 2 A schematic diagram of a spatial intelligent multimodal scene memory method provided in another embodiment of this application is shown; Figure 3 A schematic diagram of the spatial intelligent multimodal scene memory device provided in an embodiment of this application is shown; Figure 4 A schematic diagram of an electronic device for the spatial intelligent multimodal scene memory method provided in an embodiment of this application is shown. Detailed Implementation

[0019] Embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0020] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0021] This application provides a spatial intelligent multimodal scene memory method. Please refer to... Figure 1 This figure is a schematic diagram of the first embodiment of this application. The following is in conjunction with... Figure 1 The spatial intelligent multimodal scene memory method 100 provided in the first embodiment of this application will be described in detail.

[0022] Step S102: Obtain data information, that is, construct a three-dimensional scene structure in geometric space based on the multimodal data input by the sensor, wherein the three-dimensional scene structure is a discretized spatial knowledge graph.

[0023] Step S104: Construct a bidirectional constraint relationship, that is, establish a bidirectional constraint relationship between the geometrically continuous space and the symbolic discrete space for the three-dimensional scene structure.

[0024] Step S106: Establish a bidirectional mapping table. Based on the bidirectional constraint relationship, a bidirectional mapping table is established for each spatial entity, connecting the geometrically continuous space and the symbolically discrete space. This allows each spatial entity to be accessed via symbol labels or geometric coordinates recorded in the bidirectional mapping table. For example, when performing a geometric navigation task, the system prioritizes path planning and obstacle detection in the continuous space; while in semantic reasoning or target association tasks, logical retrieval and relation matching are performed through the discrete symbolic space. Bidirectional retrieval is implemented through a unified cross-spatial index framework, which establishes a bidirectional mapping table between geometric nodes and symbolic nodes, allowing each spatial entity to be accessed via symbol labels or geometric coordinates. When a user or system issues a spatial query request, the module first determines the task type and feature requirements, then performs a preliminary search in the corresponding spatial domain, and finally uses a cross-spatial consistency verification mechanism to perform geometric and semantic joint verification of the results, ensuring that the output spatial memory results are correct both logically and physically.

[0025] Step S108: Implement incremental updates, that is, based on the bidirectional mapping table, monitor changes in the geometrically continuous space and the symbolic discrete space, and maintain the bidirectional mapping table to achieve incremental updates. For example, when new perceived data arrives (e.g., point clouds collected during robot movement or new environments scanned by AR devices), the system first performs incremental updates in the geometric space, detecting changes in spatial structure, such as the appearance of new obstacles or changes in environmental topology; subsequently, these changes are projected onto the symbolic space through a consistent mapping mechanism, updating the corresponding nodes and semantic relationships. Conversely, when the symbolic space introduces new relationships due to semantic reasoning or knowledge expansion (e.g., inferring a new logical connection between two regions), the system performs structure detection and verification in the geometric space. If the relationship is reasonable, the corresponding geometric connection is added; otherwise, semantic backtracking or weight adjustment is performed. To ensure the robustness of the update process, this module introduces a joint iterative learning mechanism to perform cyclic verification and parameter optimization in both spaces, ensuring that the two spatial representations remain consistent after each update.

[0026] Step S110: Perform local reconstruction, that is, use clustering to identify the bidirectional constraint relationships in the increment, and perform local reconstruction on the geometrically continuous space and the symbolic discrete space. For example, this embodiment may include: First, new knowledge discovery: During spatial interaction, the system identifies unseen spatial structures or new semantic relationships through anomaly detection and semantic clustering; Second, adaptive reconstruction: For detected new structures, the system performs local reconstruction in the geometric space, and simultaneously generates corresponding new semantic nodes and relationships in the symbolic space; Third, consistency verification and absorption: The newly generated knowledge is verified through the cross-space consistency constraint module. If it conforms to geometric logic and semantic rules, it is incorporated into the global spatial knowledge graph.

[0027] In some embodiments, a three-dimensional scene structure is constructed in geometric space based on multimodal data input from sensors. The three-dimensional scene structure is a discretized spatial knowledge graph. This includes the following steps: First, a three-dimensional scene structure is constructed in geometric space based on multimodal data input from sensors. The multimodal data input from sensors includes point clouds, depth maps, voxel models, RGB images, and semantic tags. Second, based on semantic nodes, relational edges, and constraint rules, the directions, relationships, functions, and logical connections between spatial objects are defined, thus constructing a discretized spatial knowledge graph for the three-dimensional scene structure. For example, a three-dimensional continuous scene structure is constructed in geometric space based on multimodal sensor input (including point clouds, depth maps, voxel models, RGB images, and semantic tags). Through a multi-scale geometric coding network, spatial geometric information is transformed into a high-dimensional continuous representation to describe the topology, accessibility, and morphological features of the space. Third, in symbolic space, the system constructs a discretized spatial knowledge graph based on semantic nodes, relational edges, and constraint rules, defining the directions, relationships, functions, and logical connections between spatial objects. The two types of spaces are mapped and fused through a unified intelligent spatial semantic embedding layer: geometric features are transformed into the basis of symbolic semantic constraints through embedding mapping, while symbolic relations in turn constrain the topological connection of the geometric space, realizing the mutual association and consistent expression of continuous and discrete spaces.

[0028] In some embodiments, establishing a bidirectional constraint relationship between a geometrically continuous space and a symbolic discrete space for the 3D scene structure includes the following steps: deriving adjacency, reachability, or inclusion relationships between spatial objects based on 3D point clouds and topological information, and verifying whether these relationships are consistent with predefined relationships; when inconsistencies are found, correcting the relationships between spatial objects to ensure that geometric distance, boundary overlap rate, and spatial reachability conform to reality. For example, this embodiment is used to establish a bidirectional constraint relationship between a geometrically continuous space and a symbolic discrete space to ensure consistency between the two types of spaces in terms of structure, semantics, and logical reasoning results. In the geometry-to-symbol direction, the system derives relationships between spatial objects (such as adjacency, reachability, inclusion, etc.) based on 3D point clouds and topological information, and verifies whether these relationships are consistent with those defined in the symbolic space. When inconsistencies are found, the system corrects the symbolic space through a "spatial consistency constraint optimizer" to maintain the unity of the geometric and semantic layers. In the symbol-to-geometric direction, when new relationships or spatial hypotheses are generated in the symbolic space through logical reasoning (such as inferring that a certain region should be connected to another region), the system performs feasibility verification in the geometric space, judging its authenticity through geometric distance, boundary overlap rate, and spatial reachability.

[0029] In some embodiments, based on the bidirectional mapping table, monitoring changes in the geometrically continuous space and the symbolically discrete space, and maintaining incremental updates of the bidirectional mapping table, includes the following steps: incrementally updating the bidirectional mapping table based on the received multimodal data; and performing verification in the geometrically continuous space and the symbolically discrete space based on the incremental update, adjusting the corresponding bidirectional constraint relationships. For example, this embodiment implements a bidirectional spatial memory retrieval function in the continuous and discrete spaces of the system. Traditional spatial memory only supports unidirectional queries, while this module allows the inference engine to adaptively select whether to perform inference and verification in the continuous or discrete space according to task requirements.

[0030] In some embodiments, the system further includes: verifying the geometrically continuous space and the symbolically discrete space based on a predetermined system period, and triggering a corresponding local reconstruction mechanism when inconsistencies in the bidirectional constraint relationship occur. For example, this embodiment may include: first, dynamically monitoring the mapping accuracy, consistency, and inference accuracy of the continuous-discrete space; second, analyzing inconsistent regions through spatial feedback loops and automatically triggering a realignment or reconstruction process; and third, adjusting module parameters through adaptive learning during long-term operation to enable the system to cope with environmental changes in different scenarios and modalities.

[0031] In some embodiments, the system further includes: employing a semantic alignment regularization strategy to correct symbol labels or geometric coordinates in the geometrically continuous space and the symbolic discrete space to achieve local semantic consistency. For example, this embodiment first evaluates the semantic embedding differences between the two spaces in real time, and identifies potential inconsistencies by calculating the semantic distance matrix and logical relationship deviation. For these regions, the system employs a semantic alignment regularization strategy to adjust the node feature distribution in the continuous space and correct the semantic label relationships in the discrete space to enhance local semantic consistency.

[0032] Figure 2 Another embodiment of this disclosure is shown. Figure 2 The diagram shows: S201 represents multimodal spatial data acquisition and preprocessing. In this embodiment, a highly consistent spatial input basis is constructed through multi-source sensor fusion, providing a unified feature space for subsequent geometric and semantic modeling; S202 represents continuous-discrete dual-space representation. Through geometric continuous modeling and parallel generation of symbolic semantic graphs, the collaborative encoding and cross-domain alignment of spatial structure and semantics are achieved; S203 represents cross-spatial consistency constraints and semantic correction. A geometric-semantic bidirectional constraint mechanism is used to self-repair structural differences, ensuring the synchronization and consistency of spatial logic and physical topology; S204 represents bidirectional spatial memory retrieval and updating. It supports task-driven bidirectional spatial access and synchronous evolution, enabling the system to have real-time retrieval and dynamic learning capabilities; S205 represents spatial evolution and long-term optimization. Through periodic consistency evaluation and semantic reconstruction, self-evolving memory optimization is achieved, maintaining the long-term stability and intelligent growth of spatial cognition.

[0033] This application also provides a spatial intelligent multimodal scene memory device 300. Please see below. Figure 3 The processing flow of the spatial intelligent multimodal scene memory device 300 may include the following modules: spatial representation module 302, spatial consistency constraint module 304, bidirectional searchable space module 306, spatial joint iterative update module 308, and spatial self-learning module 310.

[0034] The spatial representation module 302 is used to construct a three-dimensional scene structure in geometric space based on multimodal data input from sensors, wherein the three-dimensional scene structure is a discretized spatial knowledge graph; The spatial consistency constraint module 304 is used to establish a two-way constraint relationship between the geometrically continuous space and the symbolic discrete space for the three-dimensional scene structure. The bidirectional searchable space module 306 is used to establish a bidirectional mapping table between geometric continuous space and symbolic discrete space for each spatial entity based on the bidirectional constraint relationship, so that each spatial entity can be accessed through the symbolic labels or geometric coordinates recorded in the bidirectional mapping table. The spatial joint iterative update module 308 is used to monitor the changes in the geometrically continuous space and the symbolic discrete space based on the bidirectional mapping table, and to maintain the bidirectional mapping table to achieve incremental updates. The spatial self-learning module 310 is used to identify the bidirectional constraint relationship in the increment by using clustering, and to perform local reconstruction on the geometrically continuous space and the symbolic discrete space.

[0035] In some embodiments, the spatial intelligent multimodal scene memory device 300 may further include: a spatial optimization module, used to verify the geometrically continuous space and the symbolic discrete space based on a predetermined system period, and to trigger a corresponding local reconstruction mechanism when the bidirectional constraint relationship is inconsistent; and a spatial semantic consistency enhancement module, used by the system to adopt a semantic alignment regularization strategy to correct symbol labels or geometric coordinates in the geometrically continuous space and the symbolic discrete space to achieve local semantic consistency.

[0036] like Figure 4 As shown, an electronic device provided in this embodiment of the invention may include a processor 420 and a memory 410. Optionally, the electronic device may further include a transceiver. The processor, memory, and transceiver may be connected via a communication bus. The memory stores computer-readable instructions, which, when executed by the processor, implement the steps of the spatial intelligent multimodal scene memory method described above.

[0037] In a specific implementation, as one example, processor 420 may include one or more CPUs.

[0038] Optionally, in a specific implementation, if the memory 410, processor 420 and communication interface 430 are integrated on a single chip, the memory 410, processor 420 and communication interface 430 can communicate with each other through an internal interface.

[0039] In a specific implementation, as one example, the electronic device may also include multiple processors, each of which may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). Here, a processor may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0040] The memory is used to store the software program that executes the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can be referred to the above method embodiment, which will not be repeated here.

[0041] A transceiver is used to communicate with network devices or with terminal devices.

[0042] Optionally, the transceiver may include a receiver and a transmitter. The receiver is used to implement the receiving function, and the transmitter is used to implement the sending function.

[0043] Optionally, the transceiver can be integrated with the processor or exist independently and coupled to the processor through the interface circuit of the electronic device. This embodiment of the invention does not specifically limit this.

[0044] It should be noted that the structure of the electronic device described above does not constitute a limitation on the electronic device. Actual electronic devices may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. Furthermore, the technical effects of the electronic device can be referred to the technical effects of the above method embodiments, and will not be repeated here.

[0045] In an exemplary embodiment, the present invention also provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the steps of the spatial intelligent multimodal scene memory method described above. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0046] This invention also provides an electronic device, which includes: a processor; and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the above-described spatial intelligent multimodal scene memory method is implemented.

[0047] This invention also provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores program code, which can be called by a processor to execute the above-described spatial intelligent multimodal scene memory method.

[0048] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0049] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0050] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0051] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0052] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0053] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0054] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A spatial intelligent multimodal scene memory method, characterized in that, include: Based on multimodal data input from sensors, a three-dimensional scene structure is constructed in geometric space, wherein the three-dimensional scene structure is a discretized spatial knowledge graph; Establish a two-way constraint relationship between the geometrically continuous space and the symbolic discrete space for the three-dimensional scene structure; Based on the aforementioned bidirectional constraint relationship, a bidirectional mapping table between geometrically continuous space and symbolic discrete space is established for each spatial entity, so that each spatial entity can be accessed through the symbolic labels or geometric coordinates recorded in the bidirectional mapping table. Based on the bidirectional mapping table, the changes in the geometrically continuous space and the symbolically discrete space are monitored, and the bidirectional mapping table is kept to achieve incremental updates. Clustering is used to identify the bidirectional constraint relationships in the increment, and local reconstruction is performed on the geometrically continuous space and the symbolic discrete space.

2. The spatial intelligent multimodal scene memory method according to claim 1, characterized in that, Based on multimodal data input from sensors, a three-dimensional scene structure is constructed in geometric space, wherein the three-dimensional scene structure is a discretized spatial knowledge graph, including the following steps: Based on multimodal data input from sensors, a three-dimensional scene structure is constructed in geometric space. The multimodal data input from sensors includes: point cloud, depth map, voxel model, RGB image and semantic label. Based on semantic nodes, relational edges, and constraint rules, the directions, relationships, functions, and logical connections between spatial objects are defined to construct a discretized spatial knowledge graph for the three-dimensional scene structure.

3. The spatial intelligent multimodal scene memory method according to claim 2, characterized in that, Establishing a bidirectional constraint relationship between the geometrically continuous space and the symbolically discrete space for the three-dimensional scene structure includes the following steps: Based on 3D point cloud and topological information, deduce the adjacency, reachability or containment relationships between spatial objects, and verify whether they are consistent with predefined relationships; When inconsistencies are found, the relationships between spatial objects are corrected to ensure that geometric distances, boundary overlap rates, and spatial accessibility are consistent with reality.

4. The spatial intelligent multimodal scene memory method according to claim 1, characterized in that, Based on the bidirectional mapping table, the changes in the geometrically continuous space and the symbolically discrete space are monitored, and the bidirectional mapping table is incrementally updated, including the following steps: Based on the received multimodal data, the bidirectional mapping table is incrementally updated; Based on incremental updates, verification is performed in the geometrically continuous space and the symbolic discrete space, and adjustments are made to the corresponding bidirectional constraint relationships.

5. The spatial intelligent multimodal scene memory method according to any one of claims 1 to 4, characterized in that, Also includes: Based on a predetermined system cycle, the geometrically continuous space and the symbolically discrete space are verified. When the bidirectional constraint relationship is inconsistent, a corresponding local reconstruction mechanism is triggered.

6. The spatial intelligent multimodal scene memory method according to claim 5, characterized in that, Also includes: The system employs a semantic alignment regularization strategy to correct symbol labels or geometric coordinates in the geometrically continuous space and the symbolically discrete space, thereby achieving local semantic consistency.

7. A spatial intelligent multimodal scene memory device, characterized in that, include: The spatial representation module is used to construct a three-dimensional scene structure in geometric space based on multimodal data input from sensors, wherein the three-dimensional scene structure is a discretized spatial knowledge graph; The spatial consistency constraint module is used to establish a two-way constraint relationship between the geometrically continuous space and the symbolic discrete space for the three-dimensional scene structure. A bidirectional searchable space module is used to establish a bidirectional mapping table between geometrically continuous space and symbolic discrete space for each spatial entity based on the bidirectional constraint relationship, so that each spatial entity can be accessed through the symbolic labels or geometric coordinates recorded in the bidirectional mapping table. The spatial joint iterative update module is used to monitor the changes in the geometrically continuous space and the symbolic discrete space based on the bidirectional mapping table, and to maintain the bidirectional mapping table to achieve incremental updates. The spatial self-learning module is used to identify bidirectional constraint relationships in the increment using clustering, and to perform local reconstruction of the geometrically continuous space and the symbolic discrete space.

8. The spatial intelligent multimodal scene memory device according to claim 7, characterized in that, Also includes: The spatial optimization module is used to verify the geometrically continuous space and the symbolic discrete space based on a predetermined system period. When the bidirectional constraint relationship is inconsistent, the corresponding local reconstruction mechanism is triggered. The spatial semantic consistency enhancement module is used by the system to adopt a semantic alignment regularization strategy to correct symbol labels or geometric coordinates in the geometrically continuous space and the symbol discrete space, thereby achieving local semantic consistency.

9. An electronic device, characterized in that, include: At least one processor; as well as At least one memory, including computer program code, the at least one memory and the computer program code being configured to utilize the at least one processor to cause the electronic device to perform the spatial intelligent multimodal scene memory method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code, which can be called by a processor to execute the spatial intelligent multimodal scene memory method as described in any one of claims 1 to 6.

Images