Multi-mode-based environment perception and control method and system, medium and product
Through multimodal data fusion and semantic understanding of large language models, the one-sided problem of environmental perception caused by single modal data is solved, intelligent control of complex environments is achieved, and the comprehensiveness and accuracy of perception and control are improved.
Patent Information
- Application Number
- CN202411056958.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-26
AI Technical Summary
The existing environment perception and control systems mainly rely on single mode data, which leads to one-sided perception of the environment and makes it difficult to achieve intelligent control and interaction in complex environments.
Multimodal data fusion technology is adopted to collect environmental data and point cloud data, preprocess and feature fusion, combine multimodal large language model for semantic understanding, and generate intelligent control instructions to achieve intelligent control.
It improves the comprehensiveness and accuracy of environmental perception, enhances the intelligence level of the system, can more accurately reflect actual needs, and realizes intelligent control of complex environments.
Smart Images

Figure CN119025850A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of multimodal data fusion, and in particular to a multimodal-based environmental perception and control method, a multimodal-based environmental perception and control method system, a multimodal-based environmental perception and control method device, a storage medium and a computer program product. Background Art
[0002] Currently, existing environmental perception and control systems mainly rely on single-modal data (such as separate vision, audio or text) for processing and analysis, and are therefore unable to integrate data from multiple perception modalities, resulting in one-sided environmental perception and difficulty in achieving intelligent control and interaction in complex environments. Summary of the invention
[0003] The main purpose of this application is to provide a multimodal environmental perception and control method, a multimodal environmental perception and control method system, a multimodal environmental perception and control method device, a storage medium and a computer program product, aiming to solve the technical problem of the one-sidedness of environmental perception.
[0004] To achieve the above objectives, the present application proposes a multi-modal environment perception and control method, which includes:
[0005] Collect environmental data and point cloud data of the preset environment, and receive user command data;
[0006] Preprocessing the environment data and the user instruction data to obtain processed data;
[0007] Performing data alignment and feature fusion on the processed data and the pre-processed point cloud data to obtain multimodal data;
[0008] The multimodal data is semantically understood according to the multimodal large language model, and intelligent control instructions are generated according to the semantic understanding results, and the preset environment is intelligently controlled through the intelligent control instructions.
[0009] In one embodiment, the step of aligning and fusing the processed data and the pre-processed point cloud data to obtain multimodal data includes:
[0010] Performing data alignment on the processed data and the pre-processed point cloud data;
[0011] Performing feature extraction on the processed data after data alignment and the preprocessed point cloud data respectively to obtain a first feature of the processed data and a second feature of the point cloud data;
[0012] The first feature and the second feature are fused to obtain multimodal data.
[0013] In one embodiment, before the step of aligning and fusing the processed data and the pre-processed point cloud data to obtain multimodal data, the step includes:
[0014] Reconstruct the preprocessed point cloud data to obtain a three-dimensional model of the preset environment;
[0015] The step of aligning and fusing the processed data and the pre-processed point cloud data to obtain multimodal data comprises:
[0016] In the three-dimensional model, the processed data and the pre-processed point cloud data are subjected to data alignment and feature fusion to obtain multimodal data.
[0017] In one embodiment, after the step of reconstructing the preprocessed point cloud data to obtain a three-dimensional model of a preset environment, the following steps are included:
[0018] In the three-dimensional model, regions and objects in the preset environment are identified by semantically segmenting and locating the preset environment;
[0019] Areas and objects are used as control objects for intelligent control of preset environments.
[0020] In addition, to achieve the above objectives, the present application also proposes a multi-modal environment perception and control system, which includes:
[0021] A multimodal data acquisition module, used to collect environmental data and point cloud data of a preset environment, and receive user command data;
[0022] A data processing module, used for preprocessing the environment data and the user instruction data to obtain processed data;
[0023] A multimodal data fusion module, used for performing data alignment and feature fusion on the processed data and the pre-processed point cloud data to obtain multimodal data;
[0024] The semantic understanding and intelligent control module performs semantic understanding on the multimodal data according to the multimodal large language model, generates intelligent control instructions according to the semantic understanding results, and intelligently controls the preset environment through the intelligent control instructions.
[0025] In one embodiment, the multimodal environment perception and control system further includes:
[0026] The multimodal data acquisition module includes an environmental data acquisition unit, a point cloud data acquisition unit and a user instruction receiving unit;
[0027] The data processing module includes: a three-dimensional model generation unit and a semantic segmentation and positioning unit;
[0028] The multimodal data fusion module includes: a data alignment unit, a feature extraction unit and a feature fusion unit;
[0029] The semantic understanding and intelligent control module includes: a multimodal semantic understanding unit and an intelligent control instruction generation unit.
[0030] In one embodiment, the environmental data collection unit is used to collect environmental data of a preset environment;
[0031] The point cloud data acquisition unit is used to collect point cloud data of a preset environment;
[0032] The user instruction receiving unit is used to receive user instruction data;
[0033] The three-dimensional model generation unit is used to reconstruct the preprocessed point cloud data to obtain a three-dimensional model of a preset environment;
[0034] The semantic segmentation and positioning unit is used to identify the area and objects in the preset environment by performing semantic segmentation and positioning on the preset environment in the three-dimensional model; and use the area and objects as control objects for intelligent control of the preset environment;
[0035] The data alignment unit is used to align the processed data with the pre-processed point cloud data;
[0036] The feature extraction unit is used to perform feature extraction on the processed data after data alignment and the preprocessed point cloud data respectively, to obtain a first feature of the processed data and a second feature of the point cloud data;
[0037] The feature fusion unit is used to fuse the first feature and the second feature to obtain multimodal data;
[0038] The multimodal semantic understanding unit is used to perform semantic understanding on the multimodal data according to a multimodal large language model;
[0039] The intelligent control instruction generation unit is used to generate intelligent control instructions according to the semantic understanding results, and intelligently control the preset environment through the intelligent control instructions.
[0040] In addition, to achieve the above-mentioned objectives, the present application also proposes a multimodal-based environmental perception and control device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the multimodal-based environmental perception and control method as described above.
[0041] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the multi-modal environment perception and control method as described above are implemented.
[0042] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the multi-modal environment perception and control method as described above.
[0043] This application obtains multimodal data by fusing environmental data, user command data and point cloud data, which can perceive the environment from multiple angles, reduce the perception bias caused by single-modal data, and solve the one-sidedness of environmental perception; through the multimodal large language model, the multimodal data is semantically understood, and intelligent control instructions are generated according to the semantic understanding results, which can fully reflect actual needs, realize intelligent control of the environment, and improve the comprehensiveness of environmental perception. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0046] Figure 1 A flowchart of the first embodiment of the multi-modal environment perception and control method provided by the present application;
[0047] Figure 2 A flowchart diagram of a second embodiment of a multi-modal environment perception and control method provided by the present application;
[0048] Figure 3 This is a schematic diagram of the module structure of the multi-modal environment perception and control system of this application;
[0049] Figure 4 A schematic diagram of a simplified process of a multi-modal environment perception and control method according to an embodiment of the present application;
[0050] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the multi-modal environment perception and control method of this application.
[0051] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0052] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.
[0053] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.
[0054] The main solution of the embodiment of the present application is to improve the comprehensiveness and accuracy of environmental perception and enhance the intelligence level of the system through the fusion of multimodal data, 3D scene reconstruction and semantic understanding of large language models.
[0055] In this embodiment, for ease of description, the following description is made by taking a multi-modal environment perception and control system (hereinafter referred to as the system) as the execution subject.
[0056] Since existing intelligent environmental perception systems mainly rely on single-modal data (such as vision, audio or text) for processing and analysis. These systems usually use convolutional neural networks (CNN), recurrent neural networks (RNN) or other deep learning models to process a single type of data, thereby realizing functions such as object recognition and behavior detection. For example: smart home system: may use camera images for intrusion detection, and use CNN models to process images and identify intruders. For example: voice assistant (uses audio data for voice recognition and command execution, using RNN or other voice recognition models, such as Alexa used by Amazon Echo). In addition, in the field of Internet of Things (IoT), sensor networks are widely used in scenarios such as environmental monitoring, smart buildings and industrial automation. However, most of these applications rely on single-modal data and it is difficult to provide comprehensive environmental perception and intelligent control.
[0057] This application provides a solution that integrates Internet of Things technology and 3D scene reconstruction technology to build an intelligent environmental perception and control system to achieve all-round perception and intelligent control of the environment. The system built based on this multimodal large language model is suitable for scenarios such as smart homes, smart cities, and industrial automation. The model uses PointNet++ for 3D scene positioning, regional division recognition and reconstruction, and combines the multimodal large language model to process and understand the environmental data collected in real time by the Internet of Things sensor and the user's text, image and voice command input, combining user input with scene information to achieve deep semantic understanding and intelligent operation instruction generation.
[0058] It should be noted that the execution subject of this embodiment may be a system, or a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, a processor, etc. capable of realizing the above functions. The following takes the system as an example to illustrate this embodiment and the following embodiments.
[0059] Based on this, the embodiment of the present application provides a multi-modal environment perception and control method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the multi-modal environment perception and control method of the present application.
[0060] In this embodiment, the multi-modal environment perception and control method includes steps S10 to S40:
[0061] Step S10, collecting environmental data and point cloud data of a preset environment, and receiving user instruction data;
[0062] It should be noted that the preset environment refers to a pre-configured and set physical or virtual space, which can be a home, office, factory workshop, virtual reality scene, etc.
[0063] Environmental data refers to information that describes the physical characteristics and state of an environment or scene. Environmental data includes but is not limited to visual data, audio data, temperature data, humidity data, light intensity, sound level, air composition, pressure, etc. In specific application scenarios, such as robotics or autonomous vehicles, environmental data may also include the location of obstacles, terrain features, the motion state of other dynamic objects, etc. Environmental data is collected through various sensors. For example, visual data is obtained through cameras and other visual sensors to capture images and video information in the environment; audio data is collected through microphones to capture sound signals in the environment, including voice commands and ambient sounds; temperature data is obtained through temperature sensors to monitor changes in ambient temperature.
[0064] Point cloud data is a data representation in three-dimensional space. It consists of a large number of points, each of which has its coordinate position in space (usually three coordinate axes: X, Y, and Z). Point cloud data can be used to describe and reconstruct the shape, structure, and surface features of an object. This type of data is commonly used in 3D scanning, laser radar measurement, photogrammetry, and other fields. Point cloud data is collected through devices such as laser radar (LiDAR) and depth cameras.
[0065] User command data refers to the command information sent by the user to a computer system, robot, smart device, etc. through an interface (such as a keyboard, touch screen, voice command, etc.). User command data includes but is not limited to text command data, voice command data, and image command data. Among them, text command data is obtained through the text data input by the user, and the user's text instructions and needs are analyzed; voice command data collects the user's voice commands through a microphone and converts the voice into text using voice recognition technology. Image command data obtains the image or video information input by the user through a camera, and performs image recognition and analysis.
[0066] User instruction data is the basis for realizing human-computer interaction, automated control and intelligent operation. Users can guide the behavior of machines or systems according to their own needs and goals.
[0067] In addition, it should be noted that the collected environmental data in the preset environment, the collected point cloud data in the preset environment, and the received user instruction data can be real-time data. It can also be data of a certain stage in the historical period, which can be exemplarily within three hours or one day; it can also be data at the current moment, updated once a second.
[0068] It is understandable that, since step S10 collects real-time data in the environment, the latest information can be quickly acquired, so that the system can accurately perceive changes in the environment, so as to better control the environment intelligently.
[0069] Step S20, pre-processing the environment data and the user instruction data to obtain processed data;
[0070] It should be noted that the preprocessing of environmental data includes denoising, normalization and format conversion of environmental data to ensure the quality and consistency of environmental data;
[0071] Preprocessing of user information data includes cleaning and standardizing the user information data to extract information such as recognition keywords, intent understanding, etc. involved in the analysis or processing tasks.
[0072] Step S30, aligning and fusing the processed data and the pre-processed point cloud data to obtain multimodal data;
[0073] It should be noted that the preprocessing of point cloud data includes preprocessing point cloud data through PointNet++ (a deep learning model for processing point cloud data) technology, solving the problem of sample unevenness through multi-scale feature extraction methods, and being able to process disordered and sparse point cloud data. The structure of PointNet++ includes sampling layer, grouping layer and feature extraction layer, which can effectively extract local and global features of point clouds. The preprocessing steps include noise removal, point cloud registration, feature extraction, etc.
[0074] Data alignment refers to the process of synchronizing or matching data from different modalities (such as images, text, point clouds) in time and / or space to ensure that the correspondence between different data sources is correct. For example, when fusing image and point cloud data, data alignment may involve matching image pixels with 3D point coordinates in the point cloud.
[0075] Feature fusion is the process of integrating feature information extracted from different data sources or modalities into a more comprehensive representation. This is often done in multimodal tasks to take advantage of multiple information sources and improve the performance and robustness of the model.
[0076] Multimodal data refers to data from multiple different information sources, which can exist in different forms, such as text, images, audio, video, point cloud, etc. The combination of multimodal data can provide richer information than a single modality, which helps to more accurately analyze and understand and improve the comprehensiveness of environmental perception.
[0077] Step S40, semantic understanding is performed on the multimodal data according to the multimodal large language model, and intelligent control instructions are generated according to the semantic understanding results, and the preset environment is intelligently controlled through the intelligent control instructions.
[0078] It should be noted that a multimodal large language model is a deep learning model that can process and understand multiple data forms (such as text, images, and audio). This model can capture the complex relationships and semantic connections between different modal data by jointly learning the representation of cross-modal data, thereby performing well in multimodal tasks such as multimodal semantic understanding, cross-modal retrieval, and generation.
[0079] Semantic understanding is an important task in natural language processing, which aims to extract and understand the deep meaning and intention from text, speech or other modal data. Semantic understanding requires comprehensive consideration of the meaning of different modal data to form a comprehensive understanding of the scene or instruction.
[0080] In this embodiment, a multimodal large language model is used to perform deep semantic analysis on multimodal data to understand the objects, behaviors, and user intentions in the scene; specifically, real-time user command data and scene environment information (including real-time environmental data and real-time point cloud data) are combined to generate a comprehensive understanding of the preset environment.
[0081] Intelligent control instructions are instructions generated by the system based on the semantic understanding of multimodal data to control or operate intelligent devices, robots, automation systems, etc. These instructions contain specific operation commands, target parameters, and execution logic to achieve intelligent control of the preset environment.
[0082] Intelligent control refers to the process of controlling the environment using artificial intelligence and automation technology. Intelligent control can adaptively adjust the control strategy according to the environmental status, user needs and preset rules to achieve efficient, safe and convenient automated management.
[0083] In one implementation, intelligent control instructions are generated based on the processed data, and the preset environment is intelligently controlled through the intelligent control instructions.
[0084] It should be noted that intelligent control instructions can be directly generated based on the processed data, specifically including learning historical user instruction data and historical environment data, predicting user needs, generating intelligent control instructions, and controlling the preset environment in advance through the intelligent control instructions. For example: preheating or precooling the room, adjusting the lighting, etc.
[0085] In another embodiment, semantic understanding is performed on the multimodal data based on a multimodal large language model, and abnormal events in a preset environment are identified based on the semantic understanding results, and an alarm is automatically triggered based on the abnormal events.
[0086] It should be noted that after semantic understanding of multimodal data, abnormal events in the preset environment are identified based on the semantic understanding results, such as equipment failures, safety hazards, etc. Through comprehensive analysis of multimodal data, the system can detect abnormal situations in a timely manner, automatically trigger alarms or take corresponding control measures to improve the safety and reliability of the environment.
[0087] This embodiment provides a multimodal-based environmental perception and control method, which uses various sensors and data acquisition devices to collect environmental information and object three-dimensional structure data, and simultaneously receives and parses user operation commands to achieve intelligent perception, understanding and response to the environment; through data alignment and feature fusion, data of different modalities can be integrated into multimodal data, and the processing and fusion of multimodal data help to improve the system's understanding and response capabilities to complex environments; the multimodal large language model can perform semantic understanding when processing multimodal data, and generate intelligent control instructions based on the understanding results, thereby intelligently controlling the preset environment.
[0088] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can refer to the above introduction, and will not be repeated in the following. Figure 2 , the step S30 includes steps A10 to A30:
[0089] Step A10, aligning the processed data with the pre-processed point cloud data;
[0090] It should be noted that, as described in step S30, the processed data and the pre-processed point cloud data are aligned, which will not be described in detail here.
[0091] Step A20, performing feature extraction on the processed data after data alignment and the pre-processed point cloud data respectively to obtain a first feature of the processed data and a second feature of the point cloud data;
[0092] It should be noted that the processed data after data alignment and the preprocessed point cloud data are matched and synchronized in time and control, ensuring that data of different modalities can be correctly associated with each other for feature fusion.
[0093] Feature extraction is a data processing method in machine learning and deep learning. Its purpose is to extract features from preprocessed data that are helpful for model understanding or prediction. Features can be image edges, shape descriptors of point clouds, word vectors of text, etc.
[0094] The first feature refers to the feature set extracted from the processed data. These processed data can be pre-processed text, images, audio, etc., and the features are the important information or patterns carried in these data, such as keywords in text, edge or color features in images, spectral features in audio, etc.
[0095] The second feature is the feature set obtained after feature extraction of point cloud data. These features may include the geometric shape, surface texture, density distribution, etc. of the point cloud. They are abstract representations of point cloud data and are used for subsequent feature fusion.
[0096] Step A30, fusing the first feature and the second feature to obtain multimodal data.
[0097] It should be noted that multimodal feature fusion refers to combining and analyzing features from different modal data (such as text, images, and point clouds) to obtain more comprehensive information. Fusion of features from different modalities after feature extraction helps capture complementary information between different modal data, thereby improving the performance and robustness of the model;
[0098] The data after fusing the first feature and the second feature constitutes multimodal data, which combines the characteristics of processed data and point cloud data, and can provide a more comprehensive and richer information representation, which is suitable for complex tasks such as environmental perception, target detection and recognition, etc.
[0099] In this embodiment, data alignment of the processed data and the pre-processed point cloud data is a key step to ensure the spatial consistency of the two. Through precise geometric transformations, such as rotation and translation, the information of different data sources is adjusted to the same reference coordinate system to ensure that the pixels in the image and the points in the point cloud correctly correspond in three-dimensional space, which provides a basis for subsequent feature extraction and fusion. Feature extraction is performed on the processed data and point cloud data after data alignment to obtain the first feature of the processed data and the second feature of the point cloud data, respectively. The extracted first feature and the second feature are fused to obtain multimodal data. Through feature fusion, the advantages of different modal data can be used to complement each other, improve the descriptive ability of the data, and improve the accuracy of three-dimensional reconstruction, the comprehensiveness of environmental perception, and the robustness of target recognition.
[0100] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those of the first embodiment can be referred to the above description and will not be described in detail later. Before step S30, the following steps are included:
[0101] Reconstruct the preprocessed point cloud data to obtain a three-dimensional model of the preset environment;
[0102] It should be noted that 3D reconstruction is the process of recovering the 3D shape and structure of an object or environment from point cloud data. This process involves steps such as point cloud registration, filtering, upsampling and surface reconstruction.
[0103] Point cloud registration refers to the process of aligning multiple point cloud data sets from different perspectives or different time points to the same coordinate system. This process achieves data fusion and reconstruction by finding the best match or transformation between point clouds. Point cloud filtering refers to the process of removing noise points, abnormal points or redundant points from point cloud data to improve data quality. Point cloud upsampling refers to increasing the number of points in the point cloud to increase the density and detail level of the data. Point cloud upsampling technologies include interpolation-based methods, generative model-based methods, and deep learning-based methods, which aim to improve the fidelity and resolution of three-dimensional models. Surface reconstruction is the process of recovering the surface geometry of an object from point cloud data.
[0104] A 3D model is a digital representation created by 3D modeling software or 3D reconstruction algorithms that accurately describes the shape, size, and spatial relationships of an object or environment. A 3D model can be a wireframe model, surface model, or solid model.
[0105] The step of aligning and fusing the processed data and the pre-processed point cloud data to obtain multimodal data comprises:
[0106] In the three-dimensional model, the processed data and the pre-processed point cloud data are subjected to data alignment and feature fusion to obtain multimodal data.
[0107] It should be noted that in the three-dimensional model, the alignment and feature fusion of the processed data and the pre-processed point cloud data integrates multiple types of information from different sources to form a richer and more comprehensive multimodal dataset to support more efficient and accurate three-dimensional environmental scene reconstruction, target recognition, environmental perception and other applications, thereby improving the system's spatial perception capabilities.
[0108] In one implementation, change information of a preset environment is obtained according to the first feature; and a three-dimensional model of the preset environment is reconstructed according to the change information and the preprocessed point cloud data;
[0109] It should be noted that the first feature of the processed data reflects the change information of the preset environment, such as: "it's dark", "the temperature is too high", "the light is too strong"; the system reconstructs the three-dimensional model of the preset environment based on the change information and the pre-processed point cloud data, which can be dynamic reconstruction or reconstruction based on historical data, and the specific method is not limited.
[0110] In this embodiment, the preprocessed point cloud data can be reconstructed to obtain a three-dimensional model of a preset environment. In the three-dimensional model, the processed data and the preprocessed point cloud data are aligned to ensure the synchronization of multimodal data in time and space. After the data is aligned, the first feature and the second feature are extracted from the processed data and the point cloud data, respectively. The first feature and the second feature are combined through feature fusion to obtain multimodal data.
[0111] By performing data alignment and feature fusion in the 3D model, the multimodal data obtained can significantly improve the accuracy of 3D reconstruction and the comprehensiveness of environmental perception. Data alignment ensures the precise spatial correspondence of different modal data and provides an accurate geometric framework for feature fusion; feature fusion improves the data description capability and the overall performance of the system by integrating the advantages of multimodal information.
[0112] In one embodiment, after reconstructing the preprocessed point cloud data to obtain a three-dimensional model of a preset environment, steps B10 to B20 are included:
[0113] Step B10, in the three-dimensional model, identifying areas and objects in the preset environment by semantically segmenting and locating the preset environment;
[0114] It should be noted that semantic segmentation is a computer vision technology used to divide an image into multiple parts or regions and identify the specific meaning or category of each part. In a 3D model, semantic segmentation can help identify and distinguish different objects and surfaces. The process of determining the position and orientation of an object or camera in a three-dimensional space. In a preset environment, positioning can be used to understand the position of an object relative to the environment and other objects. In a 3D model, a region can refer to a specific part of space that has common characteristics or functions, and can be a wall, floor, ceiling, or any other spatial division. In a 3D model, an object is the basic element that makes up the environment and can be any three-dimensional shape, such as furniture, equipment, or other three-dimensional entities.
[0115] Step B20, taking the area and the object as the control object of the intelligent control of the preset environment.
[0116] It should be noted that a specific part or space in a preset environment, such as a room, work area or a functional area, and a three-dimensional entity in the preset environment that can be identified and operated, such as furniture, electrical appliances, decorations, etc. Objects can be combined with areas to become entity objects for control and interaction.
[0117] In one embodiment, after the system generates intelligent control instructions based on the semantic understanding results, the system receives user instruction data, processes the user instruction data, interprets the user instruction data, inputs the information contained in the instruction data into the system, and the system sends feedback results to the user to achieve real-time interaction between the system and the user.
[0118] In this embodiment, by performing semantic segmentation and positioning of the preset environment, the various regions and objects in the environment can be effectively identified and understood. Semantic segmentation helps the system understand the category to which each pixel in the image belongs, while positioning provides specific information about the object in three-dimensional space. Taking regions and objects as control objects enables the system to accurately manage and adjust them. By performing semantic segmentation and positioning of regions and objects in the preset environment, the intelligent control system can identify the specific attributes and status of each control object. In this way, the system can automatically perform control tasks based on user needs, environmental conditions or other factors. Intelligent control not only improves efficiency and comfort, but also provides users with a more personalized and responsive experience.
[0119] This application also provides a multi-modal environment perception and control system, please refer to Figure 3 , the multi-modal environment perception and control system includes:
[0120] The multimodal data acquisition module 10 is used to collect environmental data and point cloud data of a preset environment and receive user instruction data;
[0121] The data processing module 20 is used to pre-process the environment data and the user instruction data to obtain processed data;
[0122] A multimodal data fusion module 30 is used to perform data alignment and feature fusion on the processed data and the pre-processed point cloud data to obtain multimodal data;
[0123] The semantic understanding and intelligent control module 40 is used to perform semantic understanding on the multimodal data according to the multimodal large language model, and generate intelligent control instructions according to the semantic understanding results, and intelligently control the preset environment through the intelligent control instructions.
[0124] Optionally, the multimodal data fusion module 30 is further used to align the processed data with the pre-processed point cloud data;
[0125] Performing feature extraction on the processed data after data alignment and the preprocessed point cloud data respectively to obtain a first feature of the processed data and a second feature of the point cloud data;
[0126] The first feature and the second feature are fused to obtain multimodal data.
[0127] Optionally, the multimodal data fusion module 30 is further used to reconstruct the preprocessed point cloud data to obtain a three-dimensional model of a preset environment;
[0128] The step of aligning and fusing the processed data and the pre-processed point cloud data to obtain multimodal data comprises:
[0129] In the three-dimensional model, the processed data and the pre-processed point cloud data are subjected to data alignment and feature fusion to obtain multimodal data.
[0130] Optionally, the multimodal data fusion module 30 is further used to identify areas and objects in the preset environment by performing semantic segmentation and positioning on the preset environment in the three-dimensional model;
[0131] Areas and objects are used as control objects for intelligent control of preset environments.
[0132] In one embodiment, the multimodal environmental perception and control system further includes: adopting a fusion architecture of cloud and edge computing, deploying the training of the multimodal large language model on the cloud, and deploying the multimodal data acquisition module, data processing module, multimodal data fusion module and semantic understanding and intelligent control module on the edge device.
[0133] In another embodiment, the edge device protects environmental data, user command data, and point cloud data through encryption.
[0134] For example, please refer to Figure 4 , Figure 4 A brief flow chart of a multimodal environment perception and control method is provided, specifically: firstly, environmental data of a preset environment is collected through an Internet of Things sensor, user command data is received, and point cloud data of the preset environment is collected through a laser radar or other device; after preprocessing the environmental data and user command data, processed data is obtained; after preprocessing the point cloud data, preprocessed point cloud data is obtained.
[0135] After the processed data is input into the multimodal data processing module, the multimodal data processing module processes the visual, audio and text data, and performs feature extraction to obtain the first feature; after the preprocessed point cloud data is input into the 3D reconstruction module, a three-dimensional model is generated, and the three-dimensional model is semantically segmented.
[0136] The first feature and the three-dimensional model are input into the data fusion module to perform data alignment and feature fusion to obtain multimodal data.
[0137] The multimodal data is input into the semantic understanding module, and the multimodal data is semantically understood through the multimodal large language model to generate a comprehensive understanding of the preset environment and recognition of user intentions.
[0138] Based on a comprehensive understanding of the preset environment and the recognition of user intentions, specific intelligent control instructions are generated through the intelligent control module, and the intelligent control instructions are executed through the environmental regulation sub-module to achieve intelligent environmental control.
[0139] The user interaction module is used to receive real-time user command data input by the user, and input the processed user command data into the above corresponding modules. At the same time, the feedback results of the intelligent control module are sent to the user to realize real-time interaction between the system and the user.
[0140] Through the above process, the system can effectively perceive the environment, understand user intentions and perform intelligent operations, thereby improving the user experience and the intelligence level of environmental control.
[0141] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the multimodal environmental perception and control method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.
[0142] In one embodiment, the multimodal data acquisition module 10 includes an environmental data acquisition unit, a point cloud data acquisition unit, and a user instruction receiving unit;
[0143] The data processing module 20 includes: a three-dimensional model generation unit and a semantic segmentation and positioning unit;
[0144] The multimodal data fusion module 30 includes: a data alignment unit, a feature extraction unit and a feature fusion unit;
[0145] The semantic understanding and intelligent control module 40 includes: a multimodal semantic understanding unit and an intelligent control instruction generation unit.
[0146] In another embodiment, the environmental data collection unit is used to collect environmental data of a preset environment;
[0147] The point cloud data acquisition unit is used to collect point cloud data of a preset environment;
[0148] The user instruction receiving unit is used to receive user instruction data;
[0149] The three-dimensional model generation unit is used to reconstruct the preprocessed point cloud data to obtain a three-dimensional model of a preset environment;
[0150] The semantic segmentation and positioning unit is used to identify the area and objects in the preset environment by performing semantic segmentation and positioning on the preset environment in the three-dimensional model; and use the area and objects as control objects for intelligent control of the preset environment;
[0151] The data alignment unit is used to align the processed data with the pre-processed point cloud data;
[0152] The feature extraction unit is used to perform feature extraction on the processed data after data alignment and the preprocessed point cloud data respectively, to obtain a first feature of the processed data and a second feature of the point cloud data;
[0153] The feature fusion unit is used to fuse the first feature and the second feature to obtain multimodal data;
[0154] The multimodal semantic understanding unit is used to perform semantic understanding on the multimodal data according to a multimodal large language model;
[0155] The intelligent control instruction generation unit is used to generate intelligent control instructions according to the semantic understanding results, and intelligently control the preset environment through the intelligent control instructions.
[0156] The multimodal environment perception and control system provided by the present application adopts the multimodal environment perception and control method in the above-mentioned embodiment, which can solve the technical problem of the one-sidedness of environment perception. Compared with the prior art, the beneficial effects of the multimodal environment perception and control system provided by the present application are the same as the beneficial effects of the multimodal environment perception and control method provided by the above-mentioned embodiment, and the other technical features in the multimodal environment perception and control system are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.
[0157] The present application provides a multimodal environment perception and control device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multimodal environment perception and control method in the first embodiment described above.
[0158] Reference below Figure 5, which shows a schematic diagram of the structure of a multimodal environment perception and control device suitable for implementing the embodiment of the present application. The multimodal environment perception and control device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The multimodal environment perception and control device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0159] like Figure 5 As shown, the multimodal environment perception and control device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the multimodal environment perception and control device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the multimodal environment perception and control device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a multimodal environment perception and control device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.
[0160] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.
[0161] The multimodal environment perception and control device provided by the present application adopts the multimodal environment perception and control method in the above embodiment, which can solve the technical problem of one-sided environment perception. Compared with the prior art, the beneficial effects of the multimodal environment perception and control device provided by the present application are the same as the beneficial effects of the multimodal environment perception and control method provided by the above embodiment, and the other technical features in the multimodal environment perception and control device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.
[0162] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.
[0163] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0164] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the multi-modal-based environment perception and control method in the above-mentioned embodiment.
[0165] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.
[0166] The computer-readable storage medium may be included in a multi-modal environment sensing and control device; or may exist independently without being assembled into a multi-modal environment sensing and control device.
[0167] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a multimodal environment perception and control device, the multimodal environment perception and control device: collects environmental data and point cloud data of a preset environment, and receives user instruction data; preprocesses the environmental data and the user instruction data to obtain processed data; performs data alignment and feature fusion on the processed data and the preprocessed point cloud data to obtain multimodal data; performs semantic understanding on the multimodal data according to a multimodal large language model, and generates intelligent control instructions according to the semantic understanding results, and intelligently controls the preset environment through the intelligent control instructions.
[0168] Computer program code for performing the operations of the present application may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0169] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0170] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.
[0171] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned multimodal environmental perception and control method, and can solve the technical problem of the one-sidedness of environmental perception. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the multimodal environmental perception and control method provided by the above-mentioned embodiment, and will not be repeated here.
[0172] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned multi-modal environment perception and control method when executed by a processor.
[0173] The computer program product provided by this application can solve the technical problem of one-sided environmental perception. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the multimodal environmental perception and control method provided by the above embodiment, which will not be repeated here.
[0174] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.
Claims
1. A multi-modal environment perception and control method, characterized in that: The multi-modal environment perception and control method comprises: Collect environmental data and point cloud data of the preset environment, and receive user command data; Preprocessing the environment data and the user instruction data to obtain processed data; Performing data alignment and feature fusion on the processed data and the pre-processed point cloud data to obtain multimodal data; The multimodal data is semantically understood according to the multimodal large language model, and intelligent control instructions are generated according to the semantic understanding results, and the preset environment is intelligently controlled through the intelligent control instructions.
2. The multi-modal environment perception and control method according to claim 1, characterized in that: The step of aligning and fusing the processed data and the pre-processed point cloud data to obtain multimodal data comprises: Performing data alignment on the processed data and the pre-processed point cloud data; Performing feature extraction on the processed data after data alignment and the preprocessed point cloud data respectively to obtain a first feature of the processed data and a second feature of the point cloud data; The first feature and the second feature are fused to obtain multimodal data.
3. The multi-modal environment perception and control method according to claim 1, characterized in that: Before the step of aligning and fusing the processed data and the pre-processed point cloud data to obtain multimodal data, the method further comprises: Reconstruct the preprocessed point cloud data to obtain a three-dimensional model of the preset environment; The step of aligning and fusing the processed data and the pre-processed point cloud data to obtain multimodal data comprises: In the three-dimensional model, the processed data and the pre-processed point cloud data are subjected to data alignment and feature fusion to obtain multimodal data.
4. The multi-modal environment perception and control method according to claim 3, characterized in that: After the step of reconstructing the preprocessed point cloud data to obtain a three-dimensional model of the preset environment, the method further comprises: In the three-dimensional model, regions and objects in the preset environment are identified by semantically segmenting and locating the preset environment; Areas and objects are used as control objects for intelligent control of preset environments.
5. A multi-modal environment perception and control system, characterized in that: The multi-modal environment perception and control system includes: A multimodal data acquisition module, used to collect environmental data and point cloud data of a preset environment, and receive user command data; A data processing module, used for preprocessing the environment data and the user instruction data to obtain processed data; A multimodal data fusion module, used for performing data alignment and feature fusion on the processed data and the pre-processed point cloud data to obtain multimodal data; The semantic understanding and intelligent control module performs semantic understanding on the multimodal data according to the multimodal large language model, generates intelligent control instructions according to the semantic understanding results, and intelligently controls the preset environment through the intelligent control instructions.
6. The multi-modal environment perception and control system according to claim 5, characterized in that: The multimodal data acquisition module includes an environmental data acquisition unit, a point cloud data acquisition unit and a user instruction receiving unit; The data processing module includes: a three-dimensional model generation unit and a semantic segmentation and positioning unit; The multimodal data fusion module includes: a data alignment unit, a feature extraction unit and a feature fusion unit; The semantic understanding and intelligent control module includes: a multimodal semantic understanding unit and an intelligent control instruction generation unit.
7. The multi-modal environment perception and control system according to claim 6, characterized in that: The environmental data collection unit is used to collect environmental data of a preset environment; The point cloud data acquisition unit is used to collect point cloud data of a preset environment; The user instruction receiving unit is used to receive user instruction data; The three-dimensional model generation unit is used to reconstruct the preprocessed point cloud data to obtain a three-dimensional model of a preset environment; The semantic segmentation and positioning unit is used to identify the area and objects in the preset environment by performing semantic segmentation and positioning on the preset environment in the three-dimensional model; and use the area and objects as control objects for intelligent control of the preset environment; The data alignment unit is used to align the processed data with the pre-processed point cloud data; The feature extraction unit is used to perform feature extraction on the processed data after data alignment and the preprocessed point cloud data respectively, to obtain a first feature of the processed data and a second feature of the point cloud data; The feature fusion unit is used to fuse the first feature and the second feature to obtain multimodal data; The multimodal semantic understanding unit is used to perform semantic understanding on the multimodal data according to a multimodal large language model; The intelligent control instruction generation unit is used to generate intelligent control instructions according to the semantic understanding results, and intelligently control the preset environment through the intelligent control instructions.
8. A multi-modal environment perception and control device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the multi-modal environment perception and control method according to any one of claims 1 to 4.
9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the multi-modal environment perception and control method according to any one of claims 1 to 4 are implemented.
10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the multi-modal environment perception and control method according to any one of claims 1 to 4 are implemented.
Citation Information
Cited By
Action generation method, system and equipment based on multi-modal large model and medium
CN119539092A
Environment data processing method and electronic equipment
CN120256877A
Environmental data processing method and electronic equipment
CN120256877B
Multi-defense-area intelligent linkage alarm method based on AIoT gateway and related equipment
CN120431699A
Multi-zone intelligent linkage alarm method based on AIoT gateway and related equipment
CN120431699B
