Smart home scene understanding and interaction method and system based on multi-modal fusion

Abstract

The invention discloses a smart home scene understanding and interaction method and system based on multi-modal fusion, and the method comprises the steps: converting collected voice data into text data, carrying out the content understanding of collected image / video data through a multi-modal visual large model in order to more effectively understand the behaviors and potential intentions of a user, and carrying out the interaction of the collected image / video data. Then performing information extraction and reasoning on text data converted from voice data and contents understood by the multi-modal visual large model by utilizing the strong logical reasoning ability of the large language model; then, an instruction is generated based on the content of user interaction, visual cue words are generated from the instruction, real-time segmentation of the image / video content is implemented according to the visual cue words by adopting a multi-modal visual large model, and the large language model and the multi-modal visual model are fused in the intelligent home scene user interaction process through the visual cue words; and finally, performing three-dimensional processing on the segmented image, and realizing deep fusion with an AR / MR scene based on real-time segmentation of the scene.

Description

Technical Field

[0001] The present invention belongs to the field of smart home technology, and specifically, relates to a smart home scene understanding and interaction method and system based on multimodal fusion. Background Art

[0002] With the continuous advancement of large model technology, its influence has gradually expanded to all areas of our daily life and production.

[0003] Traditional smart home intelligent interaction has some shortcomings: (1) Voice interaction and visual interaction are relatively independent. In existing smart home intelligent interaction solutions, voice recognition technology and visual recognition technology are usually processed independently by different modules. Voice and visual interactions in smart homes are often not well integrated, and there is a lack of effective data and information sharing mechanisms. When users use it, voice assistants cannot use visual information to assist in understanding instructions, and the visual system cannot use the history of voice interactions to optimize responses, resulting in an incoherent user interaction experience. For example, the voice assistant may not be able to combine visual information to better understand the user's instructions, or the visual system cannot use the history of voice interactions to optimize its responses. This independence limits the naturalness and efficiency of the interaction.

[0004] (2) Lack of in-depth understanding of the scene content. Existing smart homes often rely on preset rules and simple pattern recognition, lacking in-depth understanding of the home environment and user behavior, which may result in the system being unable to accurately predict user needs or provide relevant smart services, leading to a poor user experience. For example, the system may not be able to understand the user's true intentions in a specific situation, or be unable to learn from the user's living habits and provide personalized services. Summary of the invention

[0005] The present invention proposes a smart home scene understanding and interaction method and system based on multimodal fusion, which promotes the development and popularization of smart home technology in an efficient, accurate, safe and user-friendly manner.

[0006] The present invention is implemented by the following technical solutions: A smart home scene understanding and interaction method based on multimodal fusion is proposed, including: Collect multimodal data; Converting speech data into text data, and performing content understanding on image data and / or video data; Use a large language model to extract information and complete reasoning from text data and understanding content; Generate instructions based on the content of user interaction, and generate visual prompt words from the instructions; Use a multimodal visual large model to segment image / video content in real time based on visual cues; Perform 3D visualization processing and display based on content segmentation.

[0007] In some embodiments of the present invention, in the method, a multimodal visual large model is used to perform content understanding on image and / or video data.

[0008] In some embodiments of the present invention, the method further comprises: Video data is input into a multimodal visual large model in a multi-task manner for content understanding.

[0009] In some embodiments of the present invention, the method further comprises: After extracting information from text data and understanding content using a large language model and completing reasoning, a setting plan is made for the reasoning results based on model weights or user preferences; the setting plan is used to guide user interaction.

[0010] In some embodiments of the present invention, the method further comprises: The coordinate set of the image contour is obtained by coordinate processing based on the visual prompt words.

[0011] A smart home scene understanding and interaction system based on multimodal fusion is proposed, including: A collection unit, used for collecting multimodal data; A data reasoning unit is used to convert speech data into text data, and to understand the content of image and / or video data; extract information from the text data and the understood content using a large language model and complete reasoning; A visual prompt word generating unit, used to generate instructions based on the content of user interaction, and generate visual prompt words from the instructions; An image / video segmentation unit, which is used to segment the image / video content in real time according to visual prompt words using a multimodal visual large model; The display unit is used for performing three-dimensional visualization processing and display based on content segmentation.

[0012] In some embodiments of the present invention, the data reasoning unit uses a multimodal visual large model to perform content understanding on the image and / or video data.

[0013] In some embodiments of the present invention, the data inference unit inputs the video data into a multi-modal visual large model in a multi-task manner for content understanding.

[0014] In some embodiments of the present invention, after the data reasoning unit extracts information from text data and understanding content using a large language model and completes reasoning, it sets a plan for the reasoning results based on model weights or user preferences; the set plan is used to guide user interaction.

[0015] In some embodiments of the present invention, the image / video segmentation unit is further used to perform coordinate processing based on the visual prompt words to obtain image contours as a label set.

[0016] Compared with the prior art, the advantages and positive effects of the present invention are as follows: in the smart home scene understanding and interaction method and system based on multimodal fusion proposed by the present invention, the collected voice data is converted into text data. In order to more effectively understand user behavior and potential intentions, the collected image / video data is understood by a multimodal visual large model, and then the powerful logical reasoning ability of the large language model is used to extract and reason information from the text data converted from the voice data and the content understood by the multimodal visual large model; then, instructions are generated based on the content of user interaction, visual prompt words are generated from the instructions, and the multimodal visual large model is used to implement real-time segmentation of image / video content according to the visual prompt words. The large language model and the multimodal visual model are integrated in the user interaction process of the smart home scene through visual prompt words, which can achieve a more natural and coherent user interaction mode, thereby improving the user experience, and based on the fusion of the multimodal visual large model, the home environment and user behavior can be deeply understood, and the user's needs can be more accurately predicted, thereby providing more accurate and personalized related intelligent services; finally, the segmented image is processed in three dimensions, and the real-time segmentation based on the scene realizes deep integration with the AR / MR scene. The present invention improves the safety of the home environment and the efficiency of intelligent services, and also improves user satisfaction by optimizing the understanding of scene content, analyzing and predicting user needs in real time, and responding to the needs in a timely manner.

[0017] Other features and advantages of the present invention will become more apparent after reading the detailed description of the embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0019] Figure 1 This is a schematic diagram of the execution steps of the smart home scene understanding and interaction method based on multimodal fusion proposed by the present invention; Figure 2This is a schematic diagram of the smart home scene understanding and interaction process based on multimodal fusion proposed by the present invention; Figure 3 Schematic diagram of understanding of image content by a multimodal visual model in an embodiment of the present invention; Figure 4 Schematic diagram of understanding of video content by a multimodal visual model in an embodiment of the present invention; Figure 5 The following is a schematic diagram of the reasoning content given by using the LLM large language model in an embodiment of the present invention; Figure 6 This is one of the schematic diagrams of image segmentation based on the understanding of video content by the multimodal visual model in an embodiment of the present invention; Figure 7 The present invention corresponds to Figure 6 Schematic diagram of instructions generated in the embodiment; Figure 8 This is the second schematic diagram of image segmentation based on the understanding of video content by the multimodal visual model in an embodiment of the present invention; Fig. 9 The present invention corresponds to Figure 8 Schematic diagram of instructions generated in the embodiment; Fig.10 Schematic diagram of the visual prompt words generated in the embodiment of the present invention; Fig.11 Schematic diagram of image resolution in an embodiment of the present invention; Fig.12 for Fig.11 A schematic diagram of a coordinate system within the resolution range of the illustrated embodiment; Fig.13 The figure is a schematic diagram showing the effect of implementing image content segmentation based on visual prompt words by a multimodal visual model in an embodiment of the present invention; Fig.14 This is the system architecture of the smart home scene understanding and interaction system based on multimodal fusion proposed by the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] The present invention proposes a smart home scene understanding and interaction method based on multimodal fusion, aiming to solve the following problems: (1) Realize the integration of voice and visual interaction: Create an integrated system that can process voice commands and visual information simultaneously, seamlessly combine voice and visual interaction, realize the integration of voice and visual interaction, provide a more natural and coherent user interaction experience, and better understand the user's intentions and needs.

[0022] (2) Enhanced understanding of home environment and user behavior: Based on the fusion of multimodal large models, the system can gain a deep understanding of home environment and user behavior, enabling it to more accurately predict user needs and provide more precise and personalized intelligent services.

[0023] (3) Optimizing the understanding of scene content: Improving the system’s understanding of the user’s true intentions in specific situations, and providing more accurate services by analyzing the user’s behavior patterns, environmental factors, and historical interaction data.

[0024] The background knowledge of the present invention is briefly described below.

[0025] 1. Multimodal visual big model: An AI model that combines deep learning technology and large-scale data pre-training to understand and process data from multiple information sources. These information sources usually include but are not limited to images, videos, text, audio and other modalities. The goal of the model is to achieve joint understanding and analysis of different types of data to provide richer and more accurate information representation and reasoning capabilities.

[0026] Key features of the multimodal vision model include: (1) Modal fusion: The model can process and integrate multiple types of data inputs so that different modalities can complement and enhance each other’s information.

[0027] (2) Large-scale pre-training: Pre-training is usually performed on a large amount of diverse data to learn common features and representations across modalities.

[0028] (3) Contextual understanding: The model not only understands individual data points, but also understands the meaning of data points in a larger context.

[0029] (4) Flexibility: It can adapt to a variety of tasks, including classification, detection, question answering, content generation, etc.

[0030] (5) High-level representation: Through deep learning networks, such as Transformer, the model can learn high-level abstract representations of data.

[0031] (6) Attention mechanism: Attention mechanism is often used to identify and focus on the most important information in different modalities.

[0032] Multimodal visual large models have potential in various applications, such as augmented reality, autonomous driving, smart home, content recommendation, robot interaction, etc., and they can provide a more natural and intelligent user experience.

[0033] Common multimodal vision models include: (1) CLIP: The CLIP (Contrastive Language–Image Pre-training) model can understand and match images and text.

[0034] (2) GPT-4o: GPT-4 Omni (GPT4o) is a large multimodal model that can process audio, video, text, and image inputs at real-time speed and generate outputs in these modalities.

[0035] (3) Florence-2: It is a large multimodal vision model that can handle various vision tasks such as image description, object detection, visual localization, and image segmentation. Florence-2 handles these tasks using a unified framework and is trained on a large-scale dataset FLD-5B, which contains 5.4 billion annotations on 126 million images. This enables Florence-2 to demonstrate excellent performance on a variety of vision tasks, including zero-shot tasks, that is, tasks that are performed without specific task training.

[0036] (4) MiniCPM-V: This is a large multimodal model that demonstrates strong capabilities in image and video understanding. MiniCPM-V 2.6 surpasses models such as GPT-4V in single image understanding, and for the first time supports real-time video understanding on end devices such as iPad. This model is known for its efficient reasoning and low memory usage, and has strong OCR capabilities and multi-language support.

[0037] 2. Prompt word technology.

[0038] Prompt technology is the process of creating prompts or guiding the output of a language model like ChatGPT, which allows users to control the output of the model and generate text that meets their specific needs.

[0039] A good question (prompt word) should at least include four key elements: task, requirement, instruction (constraint rules or conditions), and role.

[0040] (1) Task: ChatGPT is required to complete a task and generate content that is clearly and concisely presented.

[0041] (2) Requirements: The background requirements for the task.

[0042] (3) Instructions: (Constraints or conditions) Instructions that ChatGPT should follow when generating text, satisfying certain conditions or rules to prevent it from being too divergent (4) Role: The role that the model should play when generating text, which can be professional, emotional, or identity-based.

[0043] 3. Visual Prompting project.

[0044] Visual Prompting is a technique used in the field of artificial intelligence, especially in multimodal learning and visual-language models. It involves using visual information (such as images, video frames) to guide or prompt the model to perform a specific task. This method is often used to enhance the model's understanding of visual content and improve its performance in multimodal tasks. Visual Prompt generally acts directly on the visual model and has some visual features. Its role is to help the model predict the expected answer without changing the model parameters. This method of using Prompt to modify the input space to migrate the model has the following advantages: (1) The input is generally well interpretable, and end users can intervene in the input and change the model output without obtaining the model parameters.

[0045] (2) It has good inheritance, and the model can implement a new task in the inference phase without retraining.

[0046] Visual Prompting often appears in the field of computer vision and image processing, especially in the task of object detection. This process involves identifying and locating objects in an image and generating a bounding box for each detected object. The actual coordinate data format of the detection is:<class’list’> , is a series of coordinate points, each of which consists of two numbers, representing the x and y coordinates, and is used to specify the position of a point on an image in image processing. Coordinate points are used to mark the segmentation boundary of an image and to define a polygonal area, where each coordinate point is a vertex of a polygon, usually ordered and connected in a certain order to form a closed contour.

[0047] Based on the above background knowledge, the present invention is implemented according to the following innovative ideas on the basis of the existing technology: (1) Integrate voice and visual interaction.

[0048] Through visual prompt words, the interaction between the language model and the visual model is integrated to improve the naturalness and efficiency of the interaction.

[0049] (2) Enhance the ability to understand the home environment and user behavior.

[0050] Based on the multimodal visual large model, we can enhance the understanding of home environment and user behavior and provide more accurate personalized services.

[0051] (3) Analyze and predict user needs in real time.

[0052] By analyzing and predicting user needs in real time, the response mechanism of the smart home system is optimized to improve user satisfaction.

[0053] (4) Real-time scene segmentation enables deep integration with AR / MR scenes. From the above perspectives, we can achieve efficient, accurate, secure and user-friendly understanding and interaction of smart home scenarios to promote the development and popularization of smart home technologies.

[0054] Combination Figure 1 and Figure 2 As shown, the smart home scene understanding and interaction method based on multimodal fusion proposed in the present invention includes the following steps: S1: Collect multimodal data.

[0055] The multimodal data here includes but is not limited to voice data, image data, video data, etc. The collected data is used for the following data information reasoning and content understanding.

[0056] S2: Convert the voice data into text data.

[0057] The collected voice data needs to be converted into text data to facilitate subsequent text information extraction, content understanding and reasoning.

[0058] S3: Perform content understanding on the image data and / or video data.

[0059] In order to more effectively understand user behaviors and potential intentions, the present invention uses a multimodal visual large model to perform in-depth content understanding of image and / or video content.

[0060] Taking the multimodal vision model MinCPM as an example, its understanding of image content can support the simultaneous input of multiple images for comprehensive content understanding and difference comparison, such as Figure 3 The model processing shown can be described as: Step 1: Send the image as input to the MiniCPM visual model for content understanding.

[0061] Step 2: Describe the requirements for MiniCPM.

[0062] For example: “Describe this scenario in detail and tell me what I should do next.”

[0063] Step 3: MiniCPM understands the content of the image and gives the output result after content understanding.

[0064] For example: "This scene shows a domestic kitchen setting with a focus on a kettle that is cooking. Given the position of the steam and kettle, the next step could be to add ingredients to the pot to cook, or if the pot is already hot, to remove the pot from the heater and prepare to move on to the next step in the cooking process".

[0065] like Figure 4 In the example shown, the process of understanding the content of the video can be described as: Step 1: Send a video of N seconds in length as input to the MiniCPM visual model for content understanding.

[0066] Step 2: Describe the requirements for MiniCPM.

[0067] For example: "Understand the video content".

[0068] Step 3: MiniCPM understands the content of the video and gives the output results after understanding the content.

[0069] For example: "This video shows the cooking process in a modern kitchen, from preparing ingredients to the final dish. Here is a detailed analysis of the steps: 1. Kitchen environment introduction: The video begins in a neat, modern kitchen with a variety of fresh vegetables and kitchen utensils on the countertop.

[0070] 2. Characters appear: A man in a striped shirt and apron appears in the picture, with his back to the camera, preparing food.

[0071] 3. Ingredients preparation: The figure takes cooking tools such as spatulas from hooks and processes ingredients on the countertop.

[0072] - The camera shows a variety of fresh vegetables, such as carrots, broccoli, bell peppers, etc., highlighting the freshness and diversity of the ingredients.

[0073] 4. Cutting ingredients: The figure uses a knife to cut a tomato, demonstrating skilled knife skills.

[0074] - In addition, the process of cutting cucumbers and preparing fish is also demonstrated.

[0075] 5. Cooking process: In different scenes, the figure can be seen continuing to process ingredients, possibly to create a complex dish.

[0076] - She laid out the ingredients on the cutting board, preparing for the next step of cooking.

[0077] 6. Finished product display: Finally, the video shows the finished dishes laid out on the table, including a fish dish, a stew and a salad, presenting a delicious and healthy meal.

[0078] The entire video shows the entire cooking process through clear images and smooth movements, emphasizing the freshness of the ingredients and the chef's cooking skills."

[0079] This video can be split into multiple pictures for display.

[0080] Videos sent to the multimodal visual model for content analysis can be segmented by duration, that is, videos without specified duration are sent to the multimodal visual model in a multi-task manner for content understanding.

[0081] S4: A large language model is used to extract information and complete reasoning on the text data obtained in S2 and the understanding content obtained in S3.

[0082] The present invention transmits the text and the understanding of the image / video obtained by S2 and S3 to the LLM large language model, and uses its powerful logical reasoning ability to integrate the information obtained from multiple parties and implement systematic processing. That is, the information is summarized and systematically processed in this step.

[0083] S5: Generate instructions based on the content of the user interaction.

[0084] This step is based on S4 and interacts with the user based on scene information.

[0085] The following are examples of scenarios: 1. Extract speech-to-text information: The user said, "It's time to eat, what should I do?"

[0086] 2. Extracted image / video information: The user walks into the kitchen and sees (using AR glasses or home cameras) a variety of fresh vegetables, such as carrots, broccoli, bell peppers, etc.

[0087] 3. Send these two pieces of information to the LLM large language model for reasoning, and the model gives the following Figure 5 The reasoning shown.

[0088] 4. Set the plan based on model weights or user preferences.

[0089] Suppose, after being set up, the smart home system proactively says: "I see fresh carrots, broccoli and bell peppers in the kitchen. I can recommend some healthy recipes for you, such as stir-fried pork with carrots and broccoli, or bell pepper chicken salad. How does that sound?"

[0090] The user replied, "Not bad," and then put the pot on the induction cooker to start boiling water. At the same time, the user started cutting vegetables. After a while, the water boiled. After the smart home system collected the image of the boiling water and understood the content, it actively reminded the user and said, "The pot is hot. You can remove the pot from the heater and prepare to proceed to the next step in the cooking process and put the ingredients in the pot."

[0091] Sub-scenario 1: If the user has not finished cutting vegetables at this time, the user replies: "Wait a moment, turn off the switch." After the user gives this specific instruction, the multimodal visual model will extract the next step related to the image content in the instruction. For the user's reply, the multimodal visual model will extract the "turn off the switch" said by the user and segment it on the image. Figure 6 The “off switch” in this scenario is a content-generated instruction based on user interaction, such as Figure 7 shown.

[0092] Sub-scenario 2: If the user has already put the cut ingredients next to the pot and is waiting to cook, and returns to the kitchen after hearing the prompt from the smart home system, the user looks at the stove with AR / MR glasses Figure 3 In the image scene shown in the figure, the multimodal large visual model will detect and segment the image, such as Figure 8 The user in this scenario does not have a voice response, but based on the prompt of the smart home system, "The pot is hot, you can take the pot off the heater and prepare to continue the next step of the cooking process, put the ingredients into the pot", it can be analyzed and extracted that "put the ingredients into the pot" is the content-generated instruction based on the user interaction, such as Fig. 9 shown.

[0093] S6: Generate visual prompt words from instructions.

[0094] The content generation instruction extracted in step S5 is sent to the multimodal visual model for processing, such as Figure 6 and Figure 8 As shown in the figure, the model will segment the recognized content. The image segmentation seen by the user is the final visual effect of the processing, while the multimodal visual large model directly outputs the following Fig.10 The visual prompt words shown are output as coordinate text strings in the format shown in the figure.

[0095] S7: Coordinate processing based on visual prompt words.

[0096] The data format of the visual prompt word is: Type of data:<class’list’> .like Fig.11 The resolution of the sample image is 768*512. Fig.10 As shown in , this set of data is a series of coordinate points, where each coordinate point consists of two numbers, representing the x and y coordinates respectively. In image processing, these coordinates are used to specify the position of the point on the image. Since the resolution used in the example image is 768*512, the coordinate points in this set of data are within this resolution range, and the coordinate system diagram of the size and position of the x and y coordinates is shown in 12.

[0097] For example,<loc_132><loc_565> The coordinates of a point are (132, 565), where: The first number (132) represents the x-coordinate, or horizontal position.

[0098] The second number (565) represents the y coordinate, which is the vertical position.

[0099] These coordinate points are used to mark the segmentation boundaries of the image, and they are used to define a polygonal area, where each coordinate point is a vertex of the polygon. In this case, the coordinate points are usually ordered and connected in a certain order (such as clockwise or counterclockwise) to form a closed contour.

[0100] S8: Multimodal visual large model realizes real-time segmentation of image / video content based on visual prompt words.

[0101] The image content can be segmented according to the visualization inscription (bounding contour coordinates) of step S7. The segmentation effect is as follows: Fig.13 As shown in the figure, the blue segmented area is the image content segmentation achieved through visual prompt words.

[0102] The imaging effect of the segmented image can be displayed to the user in the form of a display screen or AR / MR glasses.

[0103] S9: Perform three-dimensional visualization processing on the two-dimensional segmented image.

[0104] If the segmented two-dimensional image is to be displayed in AR / MR glasses, it needs to be processed for three-dimensional visualization. The specific processing method can refer to the existing technology.

[0105] S10: Three-dimensional display.

[0106] After content recognition, content segmentation, and 3D processing, the image is loaded into a data transmission payload that complies with 6Dof and can be used for AR / MR / VR display.

[0107] Based on the above steps, the present invention realizes the scene fusion of the large language model and the multimodal visual model through visual prompt words, and performs real-time segmentation and display processing of the scene content based on the multimodal visual model, which has the following advantages: (1) Enhanced interactive experience. The integration of voice and visual interaction provides a more natural and coherent user interaction method, thereby improving the user experience.

[0108] (2) Personalized and predictive services. The present invention can provide a more in-depth understanding of the home environment and user behavior, thereby providing more personalized services, and can predict user needs and provide corresponding intelligent services in advance.

[0109] (3) Improve safety and efficiency. By optimizing the understanding of scene content, it is possible to more accurately identify user intent and respond to requests in a timely manner, while improving the safety of the home environment and the efficiency of smart services.

[0110] The present invention also proposes a smart home scene understanding and interaction system based on multimodal fusion, such as Fig.14 As shown, including: The acquisition unit 1 is used to acquire multimodal data.

[0111] The data inference unit 2 is used to convert the speech data into text data, and to understand the content of the image and / or video data; the text data and the understood content are extracted using a large language model and the inference is completed.

[0112] The visual prompt word generating unit 3 is used to generate instructions based on the content of user interaction, and generate visual prompt words from the instructions.

[0113] The image / video segmentation unit 4 is used to segment the image / video content in real time according to the visual prompt words using a multimodal visual large model.

[0114] The display unit 5 is used for performing three-dimensional visualization processing and display based on content segmentation.

[0115] In some embodiments of the present invention, the data reasoning unit 2 uses a multimodal visual large model to perform content understanding on the image and / or video data.

[0116] In some embodiments of the present invention, the data reasoning unit 2 inputs the video data into a multi-modal visual large model in a multi-task manner for content understanding.

[0117] In some embodiments of the present invention, after the data reasoning unit 2 extracts information from the text data and the understanding content using a large language model and completes reasoning, it sets a plan for the reasoning results based on the model weight or user preference; the set plan is used to guide user interaction.

[0118] In some embodiments of the present invention, the image / video segmentation unit 4 is further configured to perform coordinate processing based on the visual prompt words to obtain image contours as a label set.

[0119] The specific process of the system realizing smart home scene understanding and interaction has been described in detail in the above method and will not be repeated here.

[0120] It should be noted that, in the specific implementation process, part of the above-mentioned method can be implemented by a hardware processor executing computer execution instructions in software form stored in the memory, which will not be elaborated here, and the programs corresponding to the actions executed can be stored in the system's computer-readable storage medium in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0121] The computer-readable storage medium mentioned above may include volatile memory, such as random access memory; may also include non-volatile memory, such as read-only memory, flash memory, hard disk or solid-state drive; may also include a combination of the above types of memory.

[0122] The processor mentioned above may also be a general term for multiple processing elements. For example, the processor may be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, field programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc., or may be a dedicated processor.

[0123] It should be pointed out that the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by ordinary technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A smart home scene understanding and interaction method based on multimodal fusion, characterized in that: include: Collect multimodal data; Converting speech data into text data, and performing content understanding on image data and / or video data; Use a large language model to extract information and complete reasoning from text data and understanding content; Generate instructions based on the content of user interaction, and generate visual prompt words from the instructions; Use a multimodal visual large model to segment image / video content in real time based on visual cues; Perform 3D visualization processing and display based on content segmentation.

2. The smart home scene understanding and interaction method based on multimodal fusion according to claim 1 is characterized in that: In the method, a multimodal visual macro model is used to perform content understanding on image and / or video data.

3. The smart home scene understanding and interaction method based on multimodal fusion according to claim 2 is characterized in that: The method further comprises: Video data is input into a multimodal visual large model in a multi-task manner for content understanding.

4. The smart home scene understanding and interaction method based on multimodal fusion according to claim 1 is characterized in that: The method further comprises: After extracting information from text data and understanding content using a large language model and completing reasoning, a setting plan is made for the reasoning results based on model weights or user preferences; the setting plan is used to guide user interaction.

5. The smart home scene understanding and interaction method based on multimodal fusion according to claim 1 is characterized in that: The method further comprises: The coordinate set of the image contour is obtained by coordinate processing based on the visual prompt words.

6. Intelligent home scene understanding and interaction system based on multimodal fusion, characterized by: include: A collection unit, used for collecting multimodal data; A data reasoning unit is used to convert speech data into text data, and to understand the content of image and / or video data; extract information from the text data and the understood content using a large language model and complete reasoning; A visual prompt word generating unit, used to generate instructions based on the content of user interaction, and generate visual prompt words from the instructions; An image / video segmentation unit, which is used to segment the image / video content in real time according to visual prompt words using a multimodal visual large model; The display unit is used for performing three-dimensional visualization processing and display based on content segmentation.

7. The smart home scene understanding and interaction system based on multimodal fusion according to claim 6 is characterized in that: The data reasoning unit uses a multimodal visual macro model to perform content understanding on image and / or video data.

8. The smart home scene understanding and interaction system based on multimodal fusion according to claim 7 is characterized in that: The data inference unit inputs the video data into a multi-modal visual large model in a multi-task manner to perform content understanding.

9. The smart home scene understanding and interaction system based on multimodal fusion according to claim 6, characterized in that: After the data reasoning unit extracts information from the text data and the understanding content using a large language model and completes reasoning, it sets a plan for the reasoning result according to the model weight or user preference; the set plan is used to guide user interaction.

10. The smart home scene understanding and interaction system based on multimodal fusion according to claim 6, characterized in that: The image / video segmentation unit is also used to perform coordinate processing based on the visual prompt words to obtain image contours as a label set.

Images