LLM-based multi-modal smart home behavior analysis method and system

Abstract

The invention provides an LLM-based multi-modal smart home behavior analysis method and system. The method comprises the steps that a smart home scene service receives a parameter data packet sent by a user terminal; calling an AI analysis module to analyze the legality of the parameter data packet; receiving alarm information sent by the intelligent camera; searching corresponding identity information and a user ID from a database according to the alarm information, and combining the alarm information, the identity information and the user ID to obtain combined information; analyzing the combined information through an AI analysis module to obtain an analysis result; receiving request parameter information sent by the user terminal; analyzing the request parameter information to obtain a matching result and a video retrieval result; the user terminal generates suggestion information according to the matching result and the video retrieval result; a large language model LLM and AI video analysis technology are adopted, real-time query and statistics of family member behaviors are obtained in a voice or text dialogue mode, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of smart home technology, and in particular to a multimodal smart home behavior analysis method and system based on LLM. Background Art

[0002] Current smart home monitoring systems mainly rely on smart cameras to record the behavior of family members in real time, and remotely view or play back the video through mobile phone applications to understand the indoor situation and the living conditions of family members.

[0003] However, the above monitoring methods have the following shortcomings: 1) Reliance on active user operations: users need to manually open the mobile application to view monitoring images or replay videos, which makes it impossible to achieve automated behavior statistics and feedback; 2) Lack of intelligent query at the semantic level: Existing systems provide information through playback or event notifications, and cannot accurately query specific behaviors. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a multimodal intelligent family behavior analysis method and system based on LLM, which combines the large language model LLM and AI video analysis technology with multimodal emotional behavior and emotion recognition functions to obtain real-time query and statistics of family members' behavior through voice or text dialogue, thereby improving user experience.

[0005] In a first aspect, an embodiment of the present invention provides a multimodal smart home behavior analysis method based on LLM, which is applied to smart home scenario services. The method includes: Receive a parameter data packet sent by a user terminal, the parameter data packet including facial information, identity information, user ID and device serial number; Invoking the AI analysis module to analyze the legitimacy of the parameter data packet and returning verification result information to the user terminal; Receive the alarm information sent by the smart camera, and generate an index number based on the URL information of the video recording in the alarm information and the device serial number; searching the corresponding identity information and the user ID from a database according to the alarm information, combining the alarm information, the identity information and the user ID to obtain combined information; Analyzing the combined information by the AI analysis module to obtain an analysis result; Receiving request parameter information sent by the user terminal, the request parameter information including the parameter data packet and a keyword; wherein the keyword is obtained by semantically parsing the input information through a large language inference model; Analyze the request parameter information to obtain matching results and video retrieval results; The matching result and the video retrieval result are sent to the user terminal, so that the user terminal generates suggestion information according to the matching result and the video retrieval result.

[0006] Furthermore, the combined information is analyzed by the AI analysis module to obtain analysis results, including: Determine the user's face database based on the user information in the combined information; Performing frame extraction processing on the video information in the combined information to obtain a face in the image; Comparing the face in the image with the faces in the face database to determine the identity of the user; The user's behavior and emotions are analyzed according to the user's identity to obtain the analysis result.

[0007] Furthermore, the user's behavior and emotions are analyzed based on the user's identity to obtain the analysis results, including: When the user's behavior is eating, the statistical number is recorded as 1; When the user's behavior is watching TV, the time of watching TV is counted and behavior description information is automatically generated.

[0008] Furthermore, the request parameter information is analyzed to obtain matching results and video retrieval results, including: Querying the database according to the request parameter information to obtain the matching result; According to the request parameter information, the video retrieval model is called to perform video retrieval to obtain the video retrieval result.

[0009] Furthermore, the AI module is called to analyze the legitimacy of the parameter data packet and return verification result information to the user terminal, including: Calling the AI module to verify the legitimacy of the facial information; When the verification is successful, a verification success response message is returned to the user terminal, and the facial information is recorded in the AI facial database; When the verification fails, verification failure response information is returned to the user terminal.

[0010] Furthermore, the method further comprises: The pop-up window module is called to return summary information of all people's behavior statistics on that day and weekly report information within a preset time to the user terminal.

[0011] In a second aspect, an embodiment of the present invention provides a multimodal smart home behavior analysis system based on LLM, which is applied to smart home scenario services. The system includes: A parameter data packet receiving module is used to receive a parameter data packet sent by a user terminal, wherein the parameter data packet includes facial information, identity information, user ID and device serial number; a legitimacy analysis module, configured to call the AI analysis module to analyze the legitimacy of the parameter data packet and return verification result information to the user terminal; An alarm information receiving module is used to receive the alarm information sent by the smart camera and generate an index number based on the URL information of the video recording in the alarm information and the device serial number; a search module, configured to search the corresponding identity information and the user ID from a database according to the alarm information, and combine the alarm information, the identity information and the user ID to obtain combined information; A first analysis module, configured to analyze the combined information through the AI analysis module to obtain an analysis result; a request parameter information receiving module, configured to receive the request parameter information sent by the user terminal, the request parameter information including the parameter data packet and a keyword; wherein the keyword is obtained by semantically parsing the input information using a large language inference model; A second analysis module is used to analyze the request parameter information to obtain matching results and video retrieval results; The sending module is used to send the matching result and the video retrieval result to the user terminal, so that the user terminal generates suggestion information according to the matching result and the video retrieval result.

[0012] Furthermore, the first analysis module is specifically configured to: Determine the user's face database based on the user information in the combined information; Performing frame extraction processing on the video information in the combined information to obtain a face in the image; Comparing the face in the image with the faces in the face database to determine the identity of the user; The user's behavior and emotions are analyzed according to the user's identity to obtain the analysis result.

[0013] In a third aspect, an embodiment of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the above-mentioned method when executing the computer program.

[0014] In a fourth aspect, an embodiment of the present invention provides a computer-readable medium having a non-volatile program code executable by a processor, wherein the program code enables the processor to execute the method as described above.

[0015] The embodiment of the present invention provides a multimodal smart home behavior analysis method and system based on LLM, including: receiving a parameter data packet sent by a user terminal, the parameter data packet including face information, identity information, user ID and device serial number; calling an AI analysis module to analyze the legitimacy of the parameter data packet and returning verification result information to the user terminal; receiving an alarm message sent by a smart camera, generating an index number from the URL information of the video recording and the device serial number in the alarm message; searching for the corresponding identity information and user ID from a database according to the alarm message, combining the alarm information, identity information and user ID to obtain combined information; and analyzing the combined information through the AI analysis module. The system analyzes the information and obtains the analysis results; receives the request parameter information sent by the user terminal, and the request parameter information includes parameter data packets and keywords; wherein the keywords are obtained by semantically parsing the input information through the large language inference model; analyzes the request parameter information to obtain matching results and video retrieval results; sends the matching results and video retrieval results to the user terminal, so that the user terminal generates suggestion information based on the matching results and video retrieval results; through the large language model LLM and AI video analysis technology, combined with multimodal emotional behavior and emotion recognition functions, obtains real-time query and statistics of family members' behaviors through voice or text dialogue, and improves user experience.

[0016] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or understood by practicing the present invention. The purposes and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0017] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 Flowchart of the multimodal smart home behavior analysis method based on LLM provided in Example 1 of the present invention; Figure 2 Schematic diagram of an application scenario of multimodal smart home behavior analysis based on LLM provided in Example 1 of the present invention; Figure 3Schematic diagram of the interaction process between a user terminal and a smart home scenario service provided in the first embodiment of the present invention; Figure 4 A schematic diagram of the interaction process between the smart camera and the smart home scene service provided in the first embodiment of the present invention; Figure 5 This is a schematic diagram of the processing process of the AI analysis module provided in Example 1 of the present invention; Figure 6 Schematic diagram of the process of processing request parameter information by the smart home scenario service provided in the first embodiment of the present invention; Figure 7 Schematic diagram of the LLM-based multimodal smart home behavior analysis system provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0021] Existing technologies rely heavily on user interaction: users need to manually open mobile apps to view surveillance footage or playback videos, making automated behavior statistics and feedback impossible. This application, however, uses natural language conversation to allow users to directly query the status and behavior statistics of family members under surveillance. This allows users to quickly access relevant information for the day without complex operations. Daily, weekly, and monthly reports also facilitate user understanding of family situations.

[0022] Existing technologies lack intelligent querying at the semantic level: existing systems can only provide information through playback or event notifications, and cannot accurately query specific behaviors based on natural language. For example, "How many times did Grandpa drink water today?" or "Did Grandpa fall today?" This application uses the LLM model to parse user semantics, extract the query object and target behavior, and quickly return behavior and emotion statistics through API calls to related services and text search and video technology, enabling more intelligent and convenient monitoring and feedback.

[0023] To facilitate understanding of this embodiment, the embodiment of the present invention is described in detail below.

[0024] Example 1: Figure 1 This is a flow chart of the LLM-based multimodal smart home behavior analysis method provided in Example 1 of the present invention.

[0025] Reference Figure 1, applied to smart home scene services, the method includes the following steps: Step S101: receiving a parameter data packet sent by a user terminal, the parameter data packet including facial information, identity information, user ID, and device serial number; Specifically, if Figure 3 As shown, the user opens the app configuration page on the user terminal and enters information into the smart home scene service, adding facial information and identity information. The user calls the smart home scene service interface and enters the facial information (face image URL) and identity information. The identity information can be "Dad" or "Grandpa," etc. The user ID and device serial number are also uploaded to the smart home scene service's face entry interface.

[0026] Step S102: Calling the AI analysis module to analyze the validity of the parameter data packet and returning verification result information to the user terminal; Step S103: Receive the alarm information sent by the smart camera, and generate an index number based on the URL information of the video recording and the device serial number in the alarm information; Specifically, refer to Figure 4 When the smart camera detects movement under the camera (such as an object moving under the camera) or recognizes a human figure, it will record the video. After the preset recording time (10 minutes), an alarm will be generated and an alarm message will be generated. The alarm information includes the device serial number, the URL information of the video recording, etc., and this information will be transmitted to the smart home scene service by calling the interface. The smart home scene service will also generate an index number based on the URL information of the video recording and the device serial number, and store the index number in the S3 cloud storage.

[0027] The Smart Home Scenario Service locates the alarm information generated by the device (smart camera) in the database and transmits the user's identity information, user ID, and other information to the AI analysis module for AI analysis. This information also includes user configuration information.

[0028] Step S104, searching the database for corresponding identity information and user ID according to the alarm information, combining the alarm information, identity information and user ID to obtain combined information; Step S105: Analyze the combined information through the AI analysis module to obtain analysis results; Step S106: Receive request parameter information sent by the user terminal, where the request parameter information includes a parameter data packet and a keyword; wherein the keyword is obtained by semantically parsing the input information using a large language inference model; Step S107: Analyze the request parameter information to obtain matching results and video retrieval results; Step S108 : sending the matching result and the video retrieval result to the user terminal, so that the user terminal generates suggestion information according to the matching result and the video retrieval result.

[0029] This application combines smart cameras and AI behavioral analysis to dynamically and quickly obtain the current behavior status of family members in the form of chat, and also provides feedback to users in daily, weekly and monthly reports, allowing users to quickly understand the situation of the day, and let users know which day to pay attention to through weekly reports.

[0030] Reference Figure 2 , mainly including smart home scene services, smart cameras, and user terminals. Database is the database, and S3 is the cloud storage database. This application combines the Large Language Model (LLM) and AI video analysis technology, integrating the large language model and camera with multimodal emotional behavior and emotion recognition capabilities. Through voice or text conversations, the LLM model can obtain real-time queries and statistics on family members' behaviors.

[0031] Furthermore, step S105 includes the following steps: Step S201, determining the user's face database based on the user information in the combined information; Step S202: extract frames from the video information in the combined information to obtain a face in the image; Step S203: Compare the face in the image with the faces in the face database to determine the user's identity; Step S204: Analyze the user's behavior and emotions based on the user's identity to obtain analysis results.

[0032] Specifically, refer to Figure 5 The AI analysis module combines the information to determine which face in the user's face database is the user's. It then extracts frames from the video and compares the face in the image with the faces in the face database to determine the user's identity. The AI analysis module returns the analysis results to the smart home scene service, which updates and accumulates the results, including emotions and behaviors, and records them in the database.

[0033] Furthermore, step S204 includes the following steps: Step S301: When the user's behavior is eating, the number of times counted is recorded as 1; Step S302: When the user's behavior is watching TV, the TV watching time is counted and behavior description information is automatically generated.

[0034] Specifically, in the process of analyzing the user's behavior and emotions, the AI analysis module analyzes the behavior and emotions of the user (for example, the mother), such as (drinking water, eating, sleeping, watching TV, or playing with a mobile phone); if it is eating, the number of times is counted as 1; if it is watching TV, the time spent watching TV is analyzed. At the same time, the AI analysis module will briefly describe the video and automatically generate a behavior description information; for example, the mother in red clothes happily watched TV for 30 minutes, and the brother in black clothes studied at the desk for 1 hour.

[0035] Furthermore, step S107 includes the following steps: Step S401, querying the database based on the request parameter information to obtain matching results; Step S402: Based on the request parameter information, the video retrieval model is called to perform video retrieval and obtain video retrieval results.

[0036] Specifically, refer to Figure 6 A voice interaction module is set up on the App of the user terminal. The user inputs text information or voice content (such as how long has my brother watched TV today, is my brother happy today), and the voice interaction module performs semantic analysis through the large language inference model to extract keywords; for example, keywords such as brother, today, TV and happy.

[0037] The voice interaction module analyzes keywords and constructs a request, combining app data with text semantics to generate request parameters and invokes the smart home scenario service interface. App data includes user information, user ID, device serial number, request object, and request content.

[0038] The smart home scenario service queries and matches the request parameters in the database, simultaneously invoking the video retrieval model to obtain video retrieval results (for example, a video of the younger brother's behavior from 10:00 to 10:15). The matching results are combined with the video retrieval results and returned to the voice interaction module, which performs inference analysis on the matching results and video retrieval results and provides relevant suggestions. For example, the younger brother watched TV for a long time today and was in a very happy mood, so he needs to protect his eyes and rest.

[0039] At the same time, the user terminal receives the statistical results of the smart home scene service and displays them in a visual interface using pie charts and bar charts. Users can click on the corresponding result to jump to the video clip for viewing. The statistical results are related to time and frequency. For example, the number of times a mother drinks water between 2 pm and 4 pm is counted.

[0040] The smart home scenario service uses an app pop-up module every night to provide a summary of everyone's daily behavior statistics to the user's terminal app. Weekly and monthly reports are also generated to provide users with information and provide reminders. This application focuses on the technical closed loop of "LLM + multimodal behavioral sentiment analysis + dynamic feedback", emphasizing the collaborative mechanism of semantic analysis and video summarization.

[0041] Furthermore, step S102 includes the following steps: Step S501: Calling the AI module to verify the legitimacy of the facial information; Step S502: When the verification is successful, a verification success response message is returned to the user terminal, and the facial information is entered into the AI facial database; Step S503: When the verification fails, a verification failure response message is returned to the user terminal.

[0042] Specifically, the smart home scene service will verify the legitimacy of the facial information and call the AI analysis module of behavioral statistics to analyze the legitimacy of the facial information; if the uploaded face is not a frontal face, a verification failure response message will be returned to the user terminal, requiring the user to upload again.

[0043] The Smart Home Scenario Service verifies the validity of the parameters. If the verification passes, the facial information is entered into the database and the AI facial database. If the configuration item already exists in the database, the record is updated. Each user corresponds to one AI facial database, which contains the person's photo and identity for subsequent facial comparison.

[0044] Furthermore, the method further comprises the following steps: Step S601: calling a pop-up window module to return summary information of all people's behavior statistics for the day and weekly report information within a preset time period to the user terminal.

[0045] This application aims to combine the Large Language Inference Model (LLM model) and AI video analysis technology, integrating the large language model and camera with multimodal emotional behavior and emotion recognition capabilities, and using the LLM model to obtain real-time queries and statistics on family members' behaviors through voice or text conversations. For example, when a user asks the question "How many times did grandpa drink water today?", the large language model can parse the query object (grandpa) and the target behavior (drinking water), and then call the smart home scene service. The system not only returns statistical data (grandpa's mood today and the number of times he drank water), but also automatically captures short video clips of related behaviors through the large text search video model, displays them through the app on the user's terminal, and provides a "View Details" option to jump to the full video.

[0046] The smart home scenario service retrieves database content and returns the corresponding behavioral statistics, and directly feeds back the statistical information to the user. In addition, this application also has the function of automatically generating weekly behavioral reports; if it detects that the elderly person's water drinking frequency is below the threshold for three consecutive days, the system will automatically push a reminder: "Grandpa has been drinking less water recently, it is recommended to pay attention"; based on activity data, the risk of falls is predicted, and health advice is included when generating weekly reports (such as "Activity level dropped sharply on Wednesday, it is recommended to check physical condition"), and behavioral pattern visualization analysis is added to the weekly / monthly reports, using timeline charts to show "Grandpa's daily peak activity time" and using heat maps to mark "Mom's most frequent activity areas."

[0047] LLM also automatically generates concise conclusions (e.g., "Grandpa's social activities increased by 20% this week; we recommend maintaining this.") If persistent negative emotions are detected (e.g., an elderly person experiencing depression for two consecutive hours), the system automatically sends an alert to the app and recommends intervention measures (e.g., "Grandpa's mood is abnormal; we recommend contacting and communicating."). By regularly summarizing family members' activities, we improve user experience and convenience.

[0048] Example 2: Figure 7 Schematic diagram of the LLM-based multimodal smart home behavior analysis system provided in Example 2 of the present invention.

[0049] Reference Figure 7 , applied to smart home scene services, the system includes: A parameter data packet receiving module is used to receive a parameter data packet sent by a user terminal, the parameter data packet including facial information, identity information, user ID and device serial number; The legality analysis module is used to call the AI analysis module to analyze the legality of the parameter data packet and return the verification result information to the user terminal; The alarm information receiving module is used to receive the alarm information sent by the smart camera and generate an index number based on the URL information of the video recording and the device serial number in the alarm information; A search module is used to search the corresponding identity information and user ID from the database according to the alarm information, and combine the alarm information, identity information and user ID to obtain combined information; The first analysis module is used to analyze the combined information through the AI analysis module to obtain analysis results; A request parameter information receiving module is configured to receive request parameter information sent by a user terminal, the request parameter information including a parameter data packet and a keyword; wherein the keyword is obtained by semantically parsing the input information using a large language inference model; The second analysis module is used to analyze the request parameter information to obtain matching results and video retrieval results; The sending module is used to send the matching results and the video retrieval results to the user terminal, so that the user terminal generates suggestion information according to the matching results and the video retrieval results.

[0050] Furthermore, the first analysis module is specifically configured to: Determine the user's face database based on the user information in the combined information; Perform frame extraction on the video information in the combined information to obtain the face in the image; Compare the face in the image with the faces in the face database to determine the user's identity; According to the user's identity, the user's behavior and emotions are analyzed to obtain analysis results.

[0051] An embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the LLM-based multimodal smart home behavior analysis method provided in the above embodiment are implemented.

[0052] An embodiment of the present invention also provides a computer-readable medium having a non-volatile program code executable by a processor, wherein a computer program is stored on the computer-readable medium, and when the computer program is run by the processor, the steps of the multimodal smart home behavior analysis method based on LLM of the above embodiment are executed.

[0053] The computer program product provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. For specific implementation, please refer to the method embodiments and will not be repeated here.

[0054] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems and devices can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0055] In addition, in the description of the embodiments of the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; they may refer to mechanical connections or electrical connections; they may refer to direct connections or indirect connections through an intermediate medium; and they may refer to internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0056] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0057] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0058] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A multimodal smart home behavior analysis method based on LLM, characterized by: Applied to smart home scenario services, the method includes: Receive a parameter data packet sent by a user terminal, the parameter data packet including facial information, identity information, user ID and device serial number; Invoking the AI analysis module to analyze the legitimacy of the parameter data packet and returning verification result information to the user terminal; Receive the alarm information sent by the smart camera, and generate an index number based on the URL information of the video recording in the alarm information and the device serial number; searching the corresponding identity information and the user ID from a database according to the alarm information, combining the alarm information, the identity information and the user ID to obtain combined information; Analyzing the combined information by the AI analysis module to obtain an analysis result; Receiving request parameter information sent by the user terminal, the request parameter information including the parameter data packet and a keyword; wherein the keyword is obtained by semantically parsing the input information through a large language inference model; Analyze the request parameter information to obtain matching results and video retrieval results; The matching result and the video retrieval result are sent to the user terminal, so that the user terminal generates suggestion information according to the matching result and the video retrieval result.

2. The multimodal smart home behavior analysis method based on LLM according to claim 1 is characterized in that: The combined information is analyzed by the AI analysis module to obtain analysis results, including: Determine the user's face database based on the user information in the combined information; Performing frame extraction processing on the video information in the combined information to obtain a face in the image; Comparing the face in the image with the faces in the face database to determine the identity of the user; The user's behavior and emotions are analyzed according to the user's identity to obtain the analysis result.

3. The multimodal smart home behavior analysis method based on LLM according to claim 2 is characterized in that: Analyzing the user's behavior and emotions based on the user's identity to obtain the analysis results includes: When the user's behavior is eating, the statistical number is recorded as 1; When the user's behavior is watching TV, the time of watching TV is counted and behavior description information is automatically generated.

4. The multimodal smart home behavior analysis method based on LLM according to claim 1 is characterized in that: Analyze the request parameter information to obtain matching results and video retrieval results, including: Querying the database according to the request parameter information to obtain the matching result; According to the request parameter information, the video retrieval model is called to perform video retrieval to obtain the video retrieval result.

5. The multimodal smart home behavior analysis method based on LLM according to claim 1 is characterized in that: Calling the AI module to analyze the legitimacy of the parameter data packet and returning verification result information to the user terminal, including: Calling the AI module to verify the legitimacy of the facial information; When the verification is successful, a verification success response message is returned to the user terminal, and the facial information is recorded in the AI facial database; When the verification fails, verification failure response information is returned to the user terminal.

6. The multimodal smart home behavior analysis method based on LLM according to claim 1 is characterized in that: The method further comprises: The pop-up window module is called to return summary information of all people's behavior statistics on that day and weekly report information within a preset time to the user terminal.

7. A multimodal intelligent home behavior analysis system based on LLM, characterized by: Applied to smart home scenario services, the system includes: A parameter data packet receiving module is used to receive a parameter data packet sent by a user terminal, wherein the parameter data packet includes facial information, identity information, user ID and device serial number; a legitimacy analysis module, configured to call the AI analysis module to analyze the legitimacy of the parameter data packet and return verification result information to the user terminal; An alarm information receiving module is used to receive the alarm information sent by the smart camera and generate an index number based on the URL information of the video recording in the alarm information and the device serial number; a search module, configured to search the corresponding identity information and the user ID from a database according to the alarm information, and combine the alarm information, the identity information and the user ID to obtain combined information; A first analysis module, configured to analyze the combined information through the AI analysis module to obtain an analysis result; a request parameter information receiving module, configured to receive the request parameter information sent by the user terminal, the request parameter information including the parameter data packet and a keyword; wherein the keyword is obtained by semantically parsing the input information using a large language inference model; A second analysis module is used to analyze the request parameter information to obtain matching results and video retrieval results; The sending module is used to send the matching result and the video retrieval result to the user terminal, so that the user terminal generates suggestion information according to the matching result and the video retrieval result.

8. The multimodal intelligent home behavior analysis system based on LLM according to claim 7 is characterized in that: The first analysis module is specifically used for: Determine the user's face database based on the user information in the combined information; Performing frame extraction processing on the video information in the combined information to obtain a face in the image; Comparing the face in the image with the faces in the face database to determine the identity of the user; The user's behavior and emotions are analyzed according to the user's identity to obtain the analysis result.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable medium having a non-volatile program code executable by a processor, characterized in that The program code causes the processor to execute the method according to any one of claims 1 to 6.

Images