A security risk warning method, system and device, electronic equipment and medium

Abstract

Embodiments of the present application provide a security risk alarm method, system, device, electronic equipment and medium, relating to the technical field of Internet of Things, the method is applied to the Internet of Things platform, the method comprises the following steps: receiving event information of a current event sent by a household device as to-be-processed event information; inputting a first prompt word and visual information in the to-be-processed event information into a preset visual large model to obtain a security risk analysis result corresponding to the current event output by the visual large model; wherein the first prompt word is used to instruct the visual large model to perform security risk analysis on the current event based on identity information in the to-be-processed event information and the visual information in the to-be-processed event information; in the case that the obtained security risk analysis result indicates that the current event exists a security risk, performing security risk alarm to an account to which the household device belongs. Through the present solution, security risk alarm can be performed more intelligently, and user experience is improved.

Description

Technical Field

[0001] This application relates to the field of Internet of Things (IoT) technology, and in particular to a method, system, device, electronic device, and medium for alerting security risks. Background Technology

[0002] Currently, in home entry scenarios, some entry devices, such as smart locks, peepholes, and doorbells, can integrate target recognition functions. To improve security, these devices can capture images outside the door for target recognition. If a preset target is detected, such as a stranger, an alarm will be triggered to the user, who can then assess whether there is a security risk.

[0003] However, this method can only identify preset targets and cannot accurately judge the security risks in the home entry scenario. It still requires users to make their own judgments, so it is not intelligent enough and results in a poor user experience. Summary of the Invention

[0004] The purpose of this application is to provide a security risk alarm method, system, device, electronic device, and medium to more intelligently perform security risk alarms. The specific technical solution is as follows:

[0005] This application first provides a security risk alarm method applied to an Internet of Things (IoT) platform, the method comprising:

[0006] The system receives event information of the current event sent by the in-home device as event information to be processed; wherein, the event information of an event includes: visual information collected by the in-home device when the event occurs, and identity information of the person identified by the in-home device when the event occurs.

[0007] The first prompt word and the visual information in the event information to be processed are input into a preset visual big model to obtain the security risk analysis result corresponding to the current event output by the visual big model; wherein, the first prompt word is used to instruct the visual big model to perform security risk analysis on the current event based on the identity information and the visual information in the event information to be processed;

[0008] If the security risk analysis results indicate that there is a security risk in the current event, a security risk alert will be sent to the account to which the in-home device belongs.

[0009] In one embodiment, the first prompt word is specifically used to instruct the visual big data model to determine whether there is a security risk in the current event based on the identity information and visual information in the event information to be processed, and to generate event text to describe the security risk of the current event; the security risk analysis result corresponding to an event includes: a judgment result indicating whether there is a security risk in the event, and event text to describe the security risk of the event.

[0010] In one embodiment, the first prompt word is specifically used to instruct the visual big model to obtain the identity information in the event information to be processed by calling the first function of the Internet of Things platform when a person is present in the scene represented by the visual information contained in the event information to be processed, and to perform a security risk analysis on the current event based on the obtained identity information and the visual information in the event information to be processed.

[0011] The method further includes:

[0012] In response to the call to the first function, the identity information in the event information to be processed is input into the visual big model;

[0013] And / or,

[0014] The first prompt word is also used to instruct the visual big data model to call a second function of the IoT platform when it identifies a security risk, so as to send a message to the IoT platform indicating that the current event has a security risk.

[0015] In one embodiment, the method further includes:

[0016] When the preset summary time is reached, the second prompt word and the event texts generated from the previous summary time to the current summary time are input into the preset large language model to obtain the first summary text of the event texts generated from the previous summary time to the current summary time, output by the large language model; wherein, the second prompt word is used to instruct the large language model to generate a summary of the input event text.

[0017] In one embodiment, the method further includes:

[0018] The event text used to describe the security risks of the current event is vectorized to obtain the vector corresponding to the current event;

[0019] Store the vector corresponding to the current event;

[0020] When an event retrieval request carrying the text to be retrieved is received from the user side, the text to be retrieved is vectorized to obtain the retrieval vector;

[0021] The vector that matches the vector to be retrieved is determined from the vectors corresponding to each historical event that are stored in advance, and is used as the target vector;

[0022] Send the event details of the target vector to the user side; wherein, the event details of a vector include: event text describing the security risks of the event corresponding to the vector, and / or, event information of the event corresponding to the vector.

[0023] In one embodiment, the method further includes:

[0024] The third prompt word and the event text describing the security risks of the event corresponding to the target vector are input into a preset large language model to obtain the second summary text output by the large language model; wherein, the third prompt word is used to instruct the large language model to generate a summary of the input event text;

[0025] The second summary text is sent to the user side.

[0026] In one embodiment, the method is applied to an in-home scenario intelligent agent in the Internet of Things platform; there is an association between the in-home device and the in-home scenario intelligent agent; the association is generated based on a subscription request sent by the account to which the in-home device belongs to the in-home device.

[0027] This application embodiment also provides a security risk alarm system, the system comprising:

[0028] An Internet of Things (IoT) platform for executing any of the security risk alerting methods described above;

[0029] The in-home device is used to send event information of the current event to the IoT platform when an event is detected.

[0030] This application also provides a security risk alarm device applied to an Internet of Things (IoT) platform, the device comprising:

[0031] The event information receiving module is used to receive event information of the current event sent by the in-home device as event information to be processed; wherein, the event information of an event includes: visual information collected by the in-home device when the event occurs, and the identity information of the person identified by the in-home device when the event occurs.

[0032] The visual big model calling module is used to input the first prompt word and the visual information in the event information to be processed into a preset visual big model, and obtain the security risk analysis result corresponding to the current event output by the visual big model; wherein, the first prompt word is used to instruct the visual big model to perform security risk analysis on the current event based on the identity information and the visual information in the event information to be processed;

[0033] The risk alarm module is used to send a security risk alarm to the account to which the in-home device belongs when the security risk analysis results indicate that there is a security risk in the current event.

[0034] In one embodiment, the first prompt word is specifically used to instruct the visual big data model to determine whether there is a security risk in the current event based on the identity information and visual information in the event information to be processed, and to generate event text to describe the security risk of the current event; the security risk analysis result corresponding to an event includes: a judgment result indicating whether there is a security risk in the event, and event text to describe the security risk of the event.

[0035] In one embodiment, the first prompt word is specifically used to instruct the visual big model to obtain the identity information in the event information to be processed by calling the first function of the Internet of Things platform when a person is present in the scene represented by the visual information contained in the event information to be processed, and to perform a security risk analysis on the current event based on the obtained identity information and the visual information in the event information to be processed.

[0036] The device further includes:

[0037] An identity information input module is used to respond to the call of the first function by inputting the identity information in the event information to be processed into the visual big model;

[0038] And / or,

[0039] The first prompt word is also used to instruct the visual big data model to call a second function of the IoT platform when it identifies a security risk, so as to send a message to the IoT platform indicating that the current event has a security risk.

[0040] In one embodiment, the device further includes:

[0041] The first summary generation module is used to input a second prompt word and the event texts generated from the previous summary time to the current summary time into a preset large language model when a preset summary time is reached, so as to obtain the first summary text of the event texts generated from the previous summary time to the current summary time output by the large language model; wherein, the second prompt word is used to instruct the large language model to generate a summary of the input event text.

[0042] In one embodiment, the device further includes:

[0043] The first vectorization module is used to vectorize the event text that describes the security risks of the current event to obtain the vector corresponding to the current event.

[0044] The vector storage module is used to store the vector corresponding to the current event;

[0045] The first vectorization module is used to vectorize the text to be retrieved when it receives an event retrieval request carrying the text to be retrieved from the user side, so as to obtain the retrieval vector.

[0046] The vector matching module is used to determine the vector that matches the vector to be retrieved from the vectors corresponding to each historical event that are stored in advance, and use it as the target vector;

[0047] The event details sending module is used to send the event details of the target vector to the user side; wherein, the event details of a vector include: event text describing the security risks of the event corresponding to the vector, and / or, event information of the event corresponding to the vector.

[0048] In one embodiment, the device further includes:

[0049] The second summary generation module is used to input a third prompt word and event text describing the security risks of the event corresponding to the target vector into a preset large language model to obtain the second summary text output by the large language model; wherein, the third prompt word is used to instruct the large language model to generate a summary of the input event text;

[0050] The second summary text is sent to the user side.

[0051] In one embodiment, the device is applied to an in-home scenario intelligent agent in the Internet of Things platform; there is an association between the in-home device and the in-home scenario intelligent agent; the association is generated based on a subscription request sent by the account to which the in-home device belongs to the in-home device.

[0052] This application also provides an electronic device, including:

[0053] Memory, used to store computer programs;

[0054] A processor, when executing a program stored in memory, implements any of the above-described methods for alerting security risks.

[0055] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-described security risk alarm methods.

[0056] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the security risk alarm methods described above.

[0057] Beneficial effects of the embodiments in this application:

[0058] The security risk alarm method provided in this application embodiment is applied to an Internet of Things (IoT) platform. It receives event information of a current event sent by an in-home device as event information to be processed. The event information includes visual information collected by the in-home device at the time the event occurs, and the identity information of the person identified by the in-home device at the time the event occurs. A first prompt word and the visual information from the event information to be processed are input into a preset visual big data model to obtain the security risk analysis result corresponding to the current event output by the visual big data model. The first prompt word is used to instruct the visual big data model to perform a security risk analysis of the current event based on the identity information and the visual information in the event information to be processed. If the obtained security risk analysis result indicates that the current event poses a security risk, a security risk alarm is sent to the account to which the in-home device belongs.

[0059] This solution utilizes the identity information of personnel identified by the in-home device and combines it with the visual big data model's ability to analyze visual information. This demonstrates that the solution integrates specific identification results with the visual big data model's ability to generalize scene recognition, thereby combining information from two dimensions to accurately analyze security risks. When the security risk analysis indicates a security risk in the current event, a security risk alert is sent to the account associated with the in-home device. Because it accurately analyzes security risks, users do not need to assess them themselves. Therefore, this solution provides more intelligent security risk alerts, improving the user experience.

[0060] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0062] Figure 1 This is a schematic diagram of a first method for alerting security risks provided in an embodiment of this application;

[0063] Figure 2 This is a schematic diagram illustrating one relationship in the security risk alarm method provided in the embodiments of this application;

[0064] Figure 3 This is a second flowchart illustrating the security risk alarm method provided in the embodiments of this application;

[0065] Figure 4 This is a schematic diagram of a third type of security risk alarm method provided in the embodiments of this application;

[0066] Figure 5 This is an interactive schematic diagram of a security risk alarm method provided in the embodiments of this application;

[0067] Figure 6 This is a schematic diagram of the fourth type of security risk alarm method provided in the embodiments of this application;

[0068] Figure 7 A schematic diagram of the system structure providing security risks for embodiments of this application;

[0069] Figure 8 A schematic diagram of the structure of the security risk alarm device provided in the embodiments of this application;

[0070] Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0071] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0072] To provide more intelligent security risk alerts for in-home scenarios, this application provides a security risk alerting method, system, device, electronic device, and medium. This method can be applied to an Internet of Things (IoT) platform. The IoT platform is used to connect and manage IoT devices, and may be, for example, a server. The method may include the following steps:

[0073] Receive event information of the current event sent by the in-home device as event information to be processed; wherein, the event information of an event includes: visual information collected by the in-home device when the event occurs, and the identity information of the person identified by the in-home device when the event occurs;

[0074] The first prompt word and the visual information in the event information to be processed are input into a preset visual big model to obtain the security risk analysis result corresponding to the current event output by the visual big model; wherein, the first prompt word is used to instruct the visual big model to perform security risk analysis on the current event based on the identity information and the visual information in the event information to be processed.

[0075] If the security risk analysis results indicate that there is a security risk in the current event, a security risk alert will be sent to the account to which the in-home device belongs.

[0076] In this embodiment, by utilizing the identity information of the person identified by the entry device and combining it with the visual big data model's ability to analyze visual information, this solution can integrate specific recognition results with the visual big data model's ability to generalize scene recognition. This allows for accurate security risk analysis by combining information from two dimensions. When the security risk analysis indicates that the current event poses a security risk, a security risk alert is sent to the account to which the entry device belongs. Because it can accurately analyze security risks, users do not need to judge the security risks themselves. Therefore, this solution can provide more intelligent security risk alerts and improve user experience.

[0077] The security risk alarm method provided in the embodiments of this application is described below with reference to the accompanying drawings. Figure 1 As shown, the method includes the following steps:

[0078] S101 receives event information of the current event sent by the in-home device as event information to be processed.

[0079] The event information for one event includes: visual information collected by the in-home device when the event occurred, and the identity information of the person identified by the in-home device when the event occurred.

[0080] The aforementioned entry-level devices refer to household appliances used in entry-level scenarios; an entry-level scenario is the situation where a person enters a room. Entry-level devices can include smart locks, peepholes, doorbells, etc. These devices can integrate target recognition and visual information acquisition functions; the visual information can be images or videos, and the devices can integrate image acquisition equipment to collect visual information. Users can pre-register identity information representing family members, such as names, and characteristic information such as facial features, into the entry-level devices, enabling the devices to identify individuals.

[0081] In one implementation, the entry device can determine that an event has occurred when it detects a person at the door. It can then identify the person at the door and capture images or videos to obtain visual information. When the identified person is a family member, the identity information included in the event information can be pre-recorded. When the identified person at the door is not a family member, the identity information included in the event information can be used to indicate that a non-family member has appeared at the door.

[0082] Of course, an event can be considered to have occurred in more than just the presence of people. For example, detecting a fire in front of the door, changes in the scene in front of the door, or the door being opened can all be considered as an event. The specific settings can be configured according to the recognition capabilities and needs of the entry-point device.

[0083] In one implementation, the method can be applied to intelligent agents in home scenarios within an IoT platform; there is an association between the home device and the intelligent agent in the home scenario; the association is generated based on the subscription request sent by the account to which the home device belongs for that home device.

[0084] In this context, an intelligent agent refers to a proxy capable of perceiving its environment and taking actions to achieve specific goals. It can be software, hardware, or a system, possessing autonomy, adaptability, and interactivity. The intelligent agent perceives changes in the environment (e.g., through sensors or data input), makes judgments and decisions based on its learned knowledge and algorithms, and then executes actions to influence the environment or achieve predetermined goals. In this embodiment, the intelligent agent for the in-home scenario can be a program or device pre-developed for the in-home scenario to execute the steps of this embodiment. This embodiment combines a scenario-based intelligent agent with a large visual model to improve the accuracy of security risk identification in in-home scenarios.

[0085] like Figure 2As shown, users can log into their accounts in advance via terminals such as smartphones and computers, and then subscribe to the services of smart agents in the home scene through the IoT platform, simultaneously associating the home device with the smart agent, i.e., adding a relationship. For example, when subscribing to the smart agent's service, a user can send the home device's identifier to the IoT platform via their terminal, allowing the IoT platform to record the association between the home device's identifier and the smart agent. Thus, when the home device detects an event, it can send event information to the IoT platform, enabling the platform to detect the smart agent associated with the home device. If an association exists between the home device and the smart agent, the platform can forward the event information to the smart agent, causing it to execute the steps in this embodiment.

[0086] S102, input the first prompt word and the visual information in the event information to be processed into the preset visual big model, and obtain the security risk analysis result corresponding to the current event output by the visual big model; wherein, the first prompt word is used to instruct the visual big model to perform security risk analysis on the current event based on the identity information and the visual information in the event information to be processed.

[0087] The aforementioned large-scale visual models, also known as visual foundation models, refer to models developed in the field of computer vision through pre-training and infrastructure modeling techniques, capable of being widely applied to various visual tasks. These models typically have a large number of parameters and strong generalization capabilities. Examples of large-scale visual models include Contrastive Language-Image Pre-training (CLE) models and Vision-Language Encoder (VLE) models.

[0088] The visual big model and the large language model described below in this solution can be deployed on an IoT platform. Of course, they can also be deployed on other platforms. In this case, the platform where the visual big model or the large language model is deployed needs to communicate with the IoT platform.

[0089] In one implementation, the first prompt word, visual information from the event information to be processed, and identity information can be input into the visual big data model together. The first prompt word may contain a statement describing the entry scene, so that the visual big data model can analyze security risks in conjunction with the scene. For example, the first prompt word may include the following: "This image was taken in front of the user's door. It is known that the person in front of the user's door is (identity information from the event information). Please carefully analyze the image and identify whether there are any fire hazards, suspicious persons loitering, abnormal accumulation of items, or other situations that may threaten personal and property safety."

[0090] In another implementation, when the visual big data model has function call functionality, the first prompt can be specifically used to instruct the visual big data model, when the visual information contained in the event information to be processed represents a person in the scene, to call the first function of the IoT platform to obtain the identity information in the event information to be processed, and to perform a security risk analysis on the current event based on the obtained identity information and the visual information in the event information to be processed; in this case, the security risk alarm method provided in this embodiment may further include:

[0091] In response to the call to the first function, the identity information from the event information to be processed is input into the visual big model.

[0092] Furthermore, the first cue word can also be used to instruct the visual big data model to call a second function of the IoT platform when it identifies a security risk, in order to send a message to the IoT platform indicating that the current event poses a security risk.

[0093] Function call capability, also known as tool use capability, refers to the ability of a large-scale model to call external tools to expand its application scope. Function call functionality allows large-scale models to directly call external functions or APIs (Application Programming Interfaces), thereby gaining the ability to perform specific tasks, acquire real-time data, or enhance decision-making. Technicians can pre-configure a first function in the IoT platform. This first function can be used to obtain identity information from event information sent by in-home devices within a certain time period prior to the current moment. This allows the identity information identified by the in-home devices to be integrated into the prompts that the visual large-scale model needs to understand. Technicians can also pre-configure a second function in the IoT platform. The visual large-scale model can send a message to the IoT platform indicating that the current event poses a security risk by calling the second function. The sent message can be a judgment result indicating that the event poses a security risk; for example, it can include the type of security risk and the degree of danger of the security risk. The type of security risk can include: fire hazard, suspicious person loitering, abnormal accumulation of items, etc.

[0094] Understandably, when there are no people in the scene represented by the visual information, or when the in-home device does not identify the presence of people in the current event, the visual big data model can also directly perform security risk analysis based on the visual information.

[0095] In one implementation, the first prompt word is specifically used to instruct the visual big data model to determine whether there is a security risk in the current event based on the identity information and visual information in the event information to be processed, and to generate event text to describe the security risk of the current event; in this case, the security risk analysis result corresponding to an event includes: a judgment result indicating whether there is a security risk in the event, and event text to describe the security risk of the event.

[0096] The first prompt word can instruct the visual big data model to generate event text describing the safety risks of an event based on the identity information and visual information within the event information to be processed. In this way, the visual big data model can intelligently generate event text regarding whether a person faces a safety risk. For example, if the visual information shows flames in the scene, and the identity information identifies a family member A, the event text generated by the visual big data model might include the following: "Family member A is at risk of fire."

[0097] For example, when the identity information represents a family member as "son," the visual model without this identity information might generate event text like: "A child wearing blue clothes, around 10 years old, comes home, looking very happy." However, using the method described in this solution, it would generate event text like: "My son, wearing blue clothes, came home, looking very happy." Clearly, this solution can generate event text more intelligently, helping users better understand the event and improving the user experience.

[0098] For example, the first prompt may include the following: "This image was captured in front of the user's house. If someone appears in the image, please call (the name of the first function) to obtain the identity information of the person appearing at the user's house. Then, based on the obtained identity information, carefully analyze the image to identify whether there are any fire hazards, suspicious persons loitering, abnormal accumulation of items, or other situations that may threaten personal and property safety. If it is determined that there is a high security risk, please call (the name of the second function). In addition, from the perspective of security risk, based on the obtained identity information, generate text to describe the current situation in front of the user's house."

[0099] S103, if the security risk analysis results indicate that there is a security risk in the current event, a security risk alert will be sent to the account to which the in-home device belongs.

[0100] For example, the IoT platform detects the network address of the terminal currently logged into by the account to which the in-home device belongs, and then sends an alarm message to the terminal based on the network address. The alarm message may include the type of security risk, the degree of danger of the security risk, and may also include event text generated by the visual big data model to describe the security risk of the event.

[0101] Alternatively, the IoT platform can directly send security risk alerts to in-home devices, enabling the devices to issue alerts via voice messages or other means. Of course, the methods for issuing security risk alerts are not limited to this.

[0102] In this embodiment, by utilizing the identity information of the person identified by the entry device and combining it with the visual big data model's ability to analyze visual information, this solution can integrate specific recognition results with the visual big data model's ability to generalize scene recognition. This allows for accurate security risk analysis by combining information from two dimensions. When the security risk analysis indicates that the current event poses a security risk, a security risk alert is sent to the account to which the entry device belongs. Because it can accurately analyze security risks, users do not need to judge the security risks themselves. Therefore, this solution can provide more intelligent security risk alerts and improve user experience.

[0103] Furthermore, since this solution does not require alarming the user every time the in-home device identifies a preset target, it also reduces the frequency of alarms to the user and reduces unnecessary disturbance to the user.

[0104] In one embodiment of this application, if the security risk analysis result corresponding to an event includes: event text describing the security risk of the event, then the security risk alarm method may further include the following steps:

[0105] When the preset summary time is reached, the second prompt word and the event texts generated from the previous summary time to the current summary time are input into the preset large language model to obtain the first summary text of the event texts generated from the previous summary time to the current summary time. The second prompt word is used to instruct the large language model to generate a summary of the input event text.

[0106] The aforementioned Large Language Model (LLM) refers to a deep learning model trained on a large amount of text data. A large language model can generate natural language text or understand the meaning of language text. Examples of large language models include GPT (Generative Pre-trained Transformer) models and DeepSeek models.

[0107] This embodiment can periodically summarize the generated event text to obtain a summary of each event sensed by the in-home device within a certain time period. For example, a summary can be performed every other day. The IoT platform can first store the obtained event text along with the event occurrence time, and then, when a summary is needed, it can retrieve the event text generated from the previous summary time to the current summary time based on the saved event occurrence time. Unlike existing technologies, this embodiment can sense and summarize the targets appearing in the events. Compared with traditional image-to-text technology, the summary content provides a better user experience due to the addition of target recognition interaction. For example, the summary content in this embodiment is similar to: "Xiaoming, the homeowner, came home 4 times today."

[0108] For example, the second prompt word could be as follows:

[0109] "The following are the events that occurred at the homeowner's doorstep: (Event text generated from the previous summary time to the current summary time). Please conduct a comprehensive summary based on these events, extracting all information related to security risks, including personal safety, property safety, and environmental safety, and covering the urgency of the risks."

[0110] After obtaining the first summary text, it can be sent to the account belonging to the in-home device. Alternatively, it can be stored on the IoT platform first, and when the user needs to view it, a summary viewing request can be sent to the IoT platform through a terminal logged into that account to retrieve the first summary text. An example first summary text is as follows:

[0111] A total of 11 events were identified today, including:

[0112] Your little Ming came home / went out 4 times (link to see details), and seems to be in a good mood.

[0113] Your partner came home / went out 3 times (link to see specific events).

[0114] You went home / out twice (link to see specific events). You got home a little late today, so please get some rest.

[0115] A stranger was detected twice (link to view specific events), once as low risk and once as high risk (you have been notified via push notification).

[0116] In this embodiment, a visual big data model generates event text describing the security risks of the current event using a first prompt word. Then, a large language model periodically summarizes the event texts generated within a period to obtain a first summary text. This first summary text can more concisely express multiple events, making it easier for users to view events that may pose security risks. Compared to existing technologies, the addition of target recognition allows for more accurate identification of security risks. For example, routine door opening by a family member is identified as no risk or low risk, while door opening or attempts by strangers are identified as high risk.

[0117] In one embodiment of this application, as Figure 3 As shown, if the security risk analysis result corresponding to the event includes: event text describing the security risk of the event, then the security risk alarm method provided in this application embodiment may further include the following steps:

[0118] S301, the event text used to describe the security risks of the current event is vectorized to obtain the vector corresponding to the current event.

[0119] In this step, the event text can be vectorized using a word embedding model, such as the Word2Vec model, or the event text can be vectorized using the large language model mentioned above.

[0120] S302 stores the vector corresponding to the current event;

[0121] The resulting vectors can be stored in a preset vector database. Additionally, the event text corresponding to the event can be stored in a preset event text database, and event information can be stored in a preset event database. The vectors, event text, and event information stored for the same event can be configured with the same index identifier, allowing for quick retrieval of the corresponding vectors, event text, and event information based on the event's index identifier.

[0122] S303: When an event retrieval request carrying the text to be retrieved is received from the user side, the text to be retrieved is vectorized to obtain the retrieval vector.

[0123] Here, the user side can refer to a terminal logged into the account belonging to the in-home device. The text to be retrieved can be natural language text input by the user on the user side through the terminal. Of course, the user can also input it via voice, in which case the IoT platform can first convert the voice into text, and then perform vectorization processing on the obtained text to obtain the vector to be retrieved. The process of vectorization processing of the text in this step can be consistent with step S301 above.

[0124] The text to be searched can be entered according to the user's search requirements. For example, if a user wants to search for events where non-family members appear in front of the door, the text to be searched can be: "Events where non-family members appear in front of the door".

[0125] Furthermore, since the event text in this solution is generated based on personnel identity information, it also provides a foundation for users to retrieve events related to specific individuals. For example, if a user wants to search for when their family member "daughter" returned home today, they can directly enter the search text: "When did my daughter return home today?" This solution will then retrieve events related to "daughter" returning home today. Therefore, because the visual model is linked to personnel identity information, this solution allows users to more easily retrieve events related to individuals, improving the user experience.

[0126] S304, determine the vector that matches the vector to be retrieved from the vectors corresponding to each historical event that are stored in advance, and use it as the target vector.

[0127] In one implementation, the similarity between the vectors corresponding to each historical event that are stored in advance and the vector to be retrieved can be calculated. For example, the cosine distance or sine distance between the vectors can be calculated. Then, the vectors corresponding to historical events with similarity greater than a threshold can be used as the target vectors. Alternatively, the vectors corresponding to a specified number of historical events with the highest similarity can be used as the target vectors.

[0128] S305, send event details of the target vector to the user side.

[0129] The event details of a vector include: event text describing the security risks of the event corresponding to the vector, and / or event information of the event corresponding to the vector.

[0130] Through this embodiment, users can retrieve historical events sensed by the in-home device using natural language to obtain the events they want to view.

[0131] In one implementation, after step S304 above, as follows: Figure 4 As shown, the alarm method for this security risk may also include:

[0132] S401, input the third prompt word and the event text used to describe the security risks of the event corresponding to the target vector into the preset large language model to obtain the second summary text output by the large language model.

[0133] The third cue word is used to instruct the large language model to generate a summary of the input event text.

[0134] The third prompt word can be similar to the second prompt word mentioned above. For example, the third prompt word could also be:

[0135] "The following are the events that occurred at the homeowner's doorstep: (Event text describing the security risks of the events corresponding to the target vector). Based on these events, please make a comprehensive summary, extract all information related to security risks, including personal safety, property safety, and environmental safety, and cover the urgency of the risks."

[0136] S402, send the event details and second summary text of the target vector to the user side.

[0137] In this embodiment, the event text describing the security risks of the current event is vectorized to obtain a vector corresponding to the current event; this vector is stored; then, when an event retrieval request carrying the text to be retrieved is received from the user, the text to be retrieved is vectorized to obtain a retrieval vector; a vector matching the retrieval vector is determined from the pre-stored vectors corresponding to each historical event, serving as the target vector; and the event details of the target vector are sent to the user, making it easier for the user to query relevant information about the event they want to obtain. Furthermore, by summarizing the event text of the retrieved events to generate a second summary text, the multiple retrieved events can be concisely expressed, making it easier for users to view and further improving the user experience.

[0138] For ease of understanding, the following description uses the security risk alarm method provided in this embodiment as an example of applying it to an intelligent agent in an in-home scenario of an IoT platform, and introduces this embodiment in conjunction with the accompanying drawings.

[0139] like Figure 5 As shown, when the entry device senses an event, such as someone appearing or opening the door, it identifies the person at the door and reports the event information to the entry scene intelligent agent.

[0140] The intelligent agent in the in-home scenario can invoke a large-scale visual model to perform image understanding on images in the event message, or video understanding on video content in the event message. This means inputting the first prompt and the visual information from the event information to be processed into a preset large-scale visual model. The first prompt can instruct the large-scale visual model to invoke a first function when the visual information in the event information indicates the presence of a person, and to invoke a second function when a security risk is detected.

[0141] The visual big data model can, upon recognizing the presence of a person in the input visual information, retrieve the person's identity through function calls, thereby parsing the event information to identify the person. It then combines the function call result to generate text, i.e., the aforementioned event text, and sends the event text to the intelligent agent in the home scene. The visual big data model can also, upon recognizing a security risk, issue a security risk alert through function calls. The intelligent agent in the home scene can then immediately issue an alert to notify the user of the existing security risk.

[0142] In the home entry scenario, the intelligent agent can send the acquired event text to the word embedding model to obtain the vector corresponding to the event text, and then store the vector and the event text.

[0143] like Figure 6 As shown, the retrieval process in this embodiment may include the following steps:

[0144] Step 1: The user inputs natural language text into the intelligent agent in the home scene.

[0145] Step 2: The intelligent agent in the home scene calls the large language model to convert the text into vectors.

[0146] Step 3: The intelligent agent in the in-home scenario matches search results from the vector database and the event text database.

[0147] That is, first query the vector database for the vector of the matching historical event, and then retrieve the event text of the historical event from the event text database.

[0148] Step 4: The intelligent agent in the in-home scenario calls the large language model to perform a summary.

[0149] That is, the third prompt word and the event text used to describe the security risks of the event corresponding to the target vector are input into a preset large language model to obtain the second summary text output by the large language model.

[0150] Step 5: The intelligent agent in the in-home scenario queries the original event based on the matching results from the vector database.

[0151] That is, retrieve event information of matching historical events from the event database.

[0152] Step 6: Return a summary and the original event list to the user.

[0153] The original event list refers to a list consisting of event information for each matching historical event.

[0154] In this embodiment, by utilizing the identity information of the person identified by the entry device and combining it with the visual big data model's ability to analyze visual information, a security risk analysis is performed on the scene in which the entry device is located. It can be seen that this solution can integrate information from multiple dimensions to accurately assess security risks. Then, when the security risk analysis results indicate that there is a security risk in the current event, a security risk alert is sent to the account to which the entry device belongs. Because it can accurately analyze security risks, users do not need to judge security risks themselves. Therefore, this solution can provide more intelligent security risk alerts and improve user experience.

[0155] In the technical solution of this application, the operations of obtaining, storing, using, processing, transmitting, providing and disclosing user personal information are all carried out with the user's authorization.

[0156] This application also provides a security risk alarm system, such as... Figure 7 As shown, the system includes:

[0157] The Internet of Things platform 701 is used for alarm methods of any of the security risks described above;

[0158] The entry-point device 702 is used to send event information of the current event to the IoT platform 701 when an event is detected. The event information includes: visual information collected by the entry-point device at the time the event occurred, and the identity information of the person identified by the entry-point device at the time the event occurred.

[0159] Based on the same inventive concept, embodiments of this application also provide a security risk alarm device, such as... Figure 8 As shown, the device includes:

[0160] The event information receiving module 801 is used to receive event information of the current event sent by the in-home device as event information to be processed; wherein, the event information of an event includes: visual information collected by the in-home device when the event occurs, and the identity information of the person identified by the in-home device when the event occurs.

[0161] The visual big model calling module 802 is used to input the first prompt word and the visual information in the event information to be processed into a preset visual big model, and obtain the security risk analysis result corresponding to the current event output by the visual big model; wherein, the first prompt word is used to instruct the visual big model to perform security risk analysis on the current event based on the identity information in the event information to be processed and the visual information in the event information to be processed;

[0162] The risk alarm module 803 is used to send a security risk alarm to the account to which the in-home device belongs when the obtained security risk analysis results indicate that there is a security risk in the current event.

[0163] In one embodiment, the first prompt word is specifically used to instruct the visual big data model to determine whether there is a security risk in the current event based on the identity information and visual information in the event information to be processed, and to generate event text to describe the security risk of the current event; the security risk analysis result corresponding to an event includes: a judgment result indicating whether there is a security risk in the event, and event text to describe the security risk of the event.

[0164] In one embodiment, the first prompt word is specifically used to instruct the visual big model to obtain the identity information in the event information to be processed by calling the first function of the IoT platform when a person is present in the scene represented by the visual information contained in the event information to be processed, and to perform a security risk analysis on the current event based on the obtained identity information and the visual information in the event information to be processed.

[0165] In one embodiment, the device further includes:

[0166] An identity information input module is used to respond to the call of the first function by inputting the identity information in the event information to be processed into the visual big model;

[0167] And / or,

[0168] The first prompt word is also used to instruct the visual big data model to call a second function of the IoT platform when it identifies a security risk, so as to send a message to the IoT platform indicating that the current event has a security risk.

[0169] In one embodiment, the device further includes:

[0170] The first summary generation module is used to input a second prompt word and the event texts generated from the previous summary time to the current summary time into a preset large language model when a preset summary time is reached, so as to obtain the first summary text of the event texts generated from the previous summary time to the current summary time output by the large language model; wherein, the second prompt word is used to instruct the large language model to generate a summary of the input event text.

[0171] In one embodiment, the device further includes:

[0172] The first vectorization module is used to vectorize the event text that describes the security risks of the current event to obtain the vector corresponding to the current event.

[0173] The vector storage module is used to store the vector corresponding to the current event;

[0174] The first vectorization module is used to vectorize the text to be retrieved when it receives an event retrieval request carrying the text to be retrieved from the user side, so as to obtain the retrieval vector.

[0175] The vector matching module is used to determine the vector that matches the vector to be retrieved from the vectors corresponding to each historical event that are stored in advance, and use it as the target vector;

[0176] The event details sending module is used to send the event details of the target vector to the user side; wherein, the event details of a vector include: event text describing the security risks of the event corresponding to the vector, and / or, event information of the event corresponding to the vector.

[0177] In one embodiment, the device further includes:

[0178] The second summary generation module is used to input a third prompt word and event text describing the security risks of the event corresponding to the target vector into a preset large language model to obtain the second summary text output by the large language model; wherein, the third prompt word is used to instruct the large language model to generate a summary of the input event text;

[0179] The second summary text is sent to the user side.

[0180] In one embodiment, the device is applied to an in-home scenario intelligent agent in the Internet of Things platform; there is an association between the in-home device and the in-home scenario intelligent agent; the association is generated based on a subscription request sent by the account to which the in-home device belongs to the in-home device.

[0181] This application also provides an electronic device, such as... Figure 9 As shown, it includes:

[0182] Memory 901 is used to store computer programs;

[0183] The processor 902, when executing a program stored in the memory 901, implements the steps of any of the above-described security risk alarm methods.

[0184] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 902, communication interface, and memory 901 communicating with each other via the communication bus.

[0185] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0186] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0187] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0188] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0189] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the security risk alarm methods described above.

[0190] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the security risk alarm methods described in the above embodiments.

[0191] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0192] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0193] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system, device, electronic device, readable storage medium, and computer program product embodiments are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0194] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for alerting security risks, characterized in that, Applied to an Internet of Things (IoT) platform, the method includes: The system receives event information of the current event sent by the in-home device as event information to be processed; wherein, the event information of an event includes: visual information collected by the in-home device when the event occurs, and identity information of the person identified by the in-home device when the event occurs. The first prompt word and the visual information in the event information to be processed are input into a preset visual big model to obtain the security risk analysis result corresponding to the current event output by the visual big model; wherein, the first prompt word is used to instruct the visual big model to perform security risk analysis on the current event based on the identity information and the visual information in the event information to be processed; If the security risk analysis results indicate that there is a security risk in the current event, a security risk alert will be sent to the account to which the in-home device belongs.

2. The method according to claim 1, characterized in that, The first prompt word is specifically used to instruct the visual big model to determine whether there is a security risk in the current event based on the identity information and visual information in the event information to be processed, and to generate event text to describe the security risk of the current event. The security risk analysis results for an event include: a judgment result indicating whether the event poses a security risk, and an event text describing the security risk of the event.

3. The method according to claim 1 or 2, characterized in that, The first prompt word is specifically used to instruct the visual big model to obtain the identity information in the event information to be processed by calling the first function of the Internet of Things platform when there is a person in the scene represented by the visual information contained in the event information to be processed, and to perform a security risk analysis on the current event based on the obtained identity information and the visual information in the event information to be processed. The method further includes: In response to the call to the first function, the identity information in the event information to be processed is input into the visual big model; And / or, The first prompt word is also used to instruct the visual big data model to call a second function of the IoT platform when it identifies a security risk, so as to send a message to the IoT platform indicating that the current event has a security risk.

4. The method according to claim 2, characterized in that, The method further includes: When the preset summary time is reached, the second prompt word and the event texts generated from the previous summary time to the current summary time are input into the preset large language model to obtain the first summary text of the event texts generated from the previous summary time to the current summary time, output by the large language model; wherein, the second prompt word is used to instruct the large language model to generate a summary of the input event text.

5. The method according to claim 2, characterized in that, The method further includes: The event text used to describe the security risks of the current event is vectorized to obtain the vector corresponding to the current event; Store the vector corresponding to the current event; When an event retrieval request carrying the text to be retrieved is received from the user side, the text to be retrieved is vectorized to obtain the retrieval vector; The vector that matches the vector to be retrieved is determined from the vectors corresponding to each historical event that are stored in advance, and is used as the target vector; Send the event details of the target vector to the user side; wherein, the event details of a vector include: event text describing the security risks of the event corresponding to the vector, and / or, event information of the event corresponding to the vector.

6. The method according to claim 5, characterized in that, The method further includes: The third prompt word and the event text describing the security risks of the event corresponding to the target vector are input into a preset large language model to obtain the second summary text output by the large language model; wherein, the third prompt word is used to instruct the large language model to generate a summary of the input event text; The second summary text is sent to the user side.

7. The method according to claim 1, characterized in that, The method is applied to the smart agent in the home scene of the Internet of Things platform; there is an association between the home device and the smart agent in the home scene; the association is generated based on the subscription request sent by the account to which the home device belongs to the home device.

8. A security risk alarm system, characterized in that, The system includes: An Internet of Things (IoT) platform for performing the method described in any one of claims 1-7; The in-home device is used to send event information of the current event to the IoT platform when an event is detected.

9. A safety risk alarm device, characterized in that, The device, applied to an Internet of Things (IoT) platform, includes: The event information receiving module is used to receive event information of the current event sent by the in-home device as event information to be processed; wherein, the event information of an event includes: visual information collected by the in-home device when the event occurs, and the identity information of the person identified by the in-home device when the event occurs. The visual big model calling module is used to input the first prompt word and the visual information in the event information to be processed into a preset visual big model, and obtain the security risk analysis result corresponding to the current event output by the visual big model; wherein, the first prompt word is used to instruct the visual big model to perform security risk analysis on the current event based on the identity information and the visual information in the event information to be processed; The risk alarm module is used to send a security risk alarm to the account to which the in-home device belongs when the security risk analysis results indicate that there is a security risk in the current event.

10. The apparatus according to claim 9, characterized in that, The first prompt word is specifically used to instruct the visual big model to determine whether there is a security risk in the current event based on the identity information and visual information in the event information to be processed, and to generate event text to describe the security risk of the current event. The security risk analysis results for an event include: a judgment result indicating whether the event poses a security risk, and an event text describing the security risk of the event; And / or, The first prompt word is specifically used to instruct the visual big model to obtain the identity information in the event information to be processed by calling the first function of the Internet of Things platform when there is a person in the scene represented by the visual information contained in the event information to be processed, and to perform a security risk analysis on the current event based on the obtained identity information and the visual information in the event information to be processed. The device further includes: An identity information input module is used to respond to the call of the first function by inputting the identity information in the event information to be processed into the visual big model; And / or, The first prompt word is also used to instruct the visual big data model to call the second function of the IoT platform when it identifies a security risk, so as to send a message to the IoT platform indicating that the current event has a security risk; And / or, The device further includes: The first summary generation module is used to input a second prompt word and the event texts generated from the previous summary time to the current summary time into a preset large language model when a preset summary time is reached, so as to obtain the first summary text of the event texts generated from the previous summary time to the current summary time output by the large language model; wherein, the second prompt word is used to instruct the large language model to generate a summary of the input event text. And / or, The device further includes: The first vectorization module is used to vectorize the event text that describes the security risks of the current event to obtain the vector corresponding to the current event. The vector storage module is used to store the vector corresponding to the current event; The first vectorization module is used to vectorize the text to be retrieved when it receives an event retrieval request carrying the text to be retrieved from the user side, so as to obtain the retrieval vector. The vector matching module is used to determine the vector that matches the vector to be retrieved from the vectors corresponding to each historical event that are stored in advance, and use it as the target vector; An event details sending module is used to send the event details of the target vector to the user side; wherein, the event details of a vector include: event text describing the security risks of the event corresponding to the vector, and / or, event information of the event corresponding to the vector; And / or, The device further includes: The second summary generation module is used to input a third prompt word and event text describing the security risks of the event corresponding to the target vector into a preset large language model to obtain the second summary text output by the large language model; wherein, the third prompt word is used to instruct the large language model to generate a summary of the input event text; Send the second summary text to the user side; And / or, The device is applied to the smart agent in the home scene of the Internet of Things platform; there is an association between the home device and the smart agent in the home scene; the association is generated based on the subscription request sent by the account to which the home device belongs to the home device.

11. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-7.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

13. A computer program product containing instructions, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method described in any one of claims 1-7.

Images