Home scene monitoring method and monitoring system supporting custom detection events

Abstract

The application relates to a home scene monitoring method supporting a self-defined detection event, which is applied to a server end of a monitoring system, the server end is configured with a multimodal large model; the monitoring system further comprises a mobile terminal and an image acquisition device in communication with the server end, the mobile terminal is configured to acquire a detection event defined by a user, the detection event comprises at least one detection object and at least one object behavior related to the detection object; the method comprises the following steps: acquiring a detection image collected by the image acquisition device; receiving the detection event sent by the mobile terminal, and determining whether the detection object is contained in the detection image by using the multimodal large model; and taking the detection image containing the detection object as a target image; determining whether the corresponding object behavior exists in the target image by using the multimodal large model, and notifying the mobile terminal if the corresponding object behavior exists. The method can realize intelligent detection based on the personalized setting of the user, and provide the user with flexible and efficient monitoring experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of smart home, in particular to a home scene monitoring method and system supporting custom detection events and an electronic device. BACKGROUND

[0002] With the development of smart home technology, more and more users install cameras in their homes for security or contact reasons and view them through mobile phones. Since users cannot check the camera at all times, they may miss many unexpected situations.

[0003] In the field of smart home, cameras are usually installed in the interior of residents' homes such as living rooms, rooms, etc. to better view the situation at home. However, the detection capability of specific behaviors of a single object is very limited, and usually only simple content such as whether a person moves, a vehicle appears, or a pet appears can be detected, and most cameras on the market can only use the monitoring settings set at the factory, which cannot meet the monitoring needs of different complex behaviors or different objects in the home scene. SUMMARY

[0004] Therefore, it is necessary to provide a home scene monitoring method and system supporting custom detection events and an electronic device to solve the above technical problems.

[0005] In a first aspect, the present application provides a home scene monitoring method supporting custom detection events, applied to a server end of a monitoring system, the server end being configured with a multi-modal large model; the monitoring system further includes a mobile terminal and an image acquisition device in communication with the server end, the mobile terminal being configured to obtain a user-defined detection event, the detection event including at least one detection object and at least one object behavior related to the detection object; the home scene monitoring method includes:

[0006] acquiring a detection image collected by the image acquisition device;

[0007] receiving the detection event sent by the mobile terminal and determining whether the detection image contains the detection object using the multi-modal large model; and taking the detection image containing the detection object as a target image;

[0008] using the multi-modal large model to determine whether the target image contains a corresponding object behavior, and if so, notifying the mobile terminal.

[0009] In one embodiment, the detection object is an object whose position coordinates in the home can change; and acquiring the detection image collected by the image acquisition device includes:

[0010] In response to the event instruction detected by the image acquisition device, a snapshot instruction is sent to the image acquisition device to make the image acquisition device save the detection image and upload the detection image to the server in response to the snapshot instruction; wherein the image acquisition device starts the movement detection function in response to the start instruction sent by the mobile terminal, and sends an event instruction to the server in the case of detecting the object movement event.

[0011] In one of the embodiments, the detected object is a pet in the home, and the object behavior is that the pet moves to a specific object in the home scene; the target image is determined to have the corresponding object behavior by using a multi-modal large model, including:

[0012] The bounding box of the pet and the bounding box of the specific object in the target image are determined based on the multi-modal model, and the multi-modal model includes a pet recognition model;

[0013] The coordinates of the two bounding boxes are obtained, and the intersection region between the two bounding boxes is determined based on the coordinates of the two bounding boxes;

[0014] If the area of the intersection region is greater than a set threshold, the target image has the object behavior.

[0015] In one of the embodiments, the detected object is a child in the home scene, and the object behavior is that the child moves to a specific object, and the specific object is an object with fixed position coordinates in the home scene; the target image is determined to have the corresponding object behavior by using a multi-modal large model, including:

[0016] The bounding box of the child and the bounding box of the specific object in the target image are determined based on the multi-modal model, and the multi-modal model includes a pet recognition model;

[0017] If there is an intersection region between the first bounding box and the second bounding box, it is determined whether the center of the first bounding box is above the center of the second bounding box;

[0018] If yes, it is determined whether the length-width ratio of the first bounding box is less than a preset ratio;

[0019] If yes, it is determined that the target image has the object behavior.

[0020] In one of the embodiments, the detection event sent by the mobile terminal includes: in response to the detection request sent by the mobile terminal, the detection request is parsed to determine the detection event sent by the mobile terminal; wherein the detection event includes the detection object and the object behavior, and the detection event also includes the period of generating the detection report; the home scene monitoring method further includes:

[0021] If the detection image contains the detection object, the server sends a notification to the mobile terminal, and when the period for generating the detection report is reached, the multi-modal large model is used to analyze the detection results of the object behavior in the target image in the period and generate a detection report, which is sent to the mobile terminal.

[0022] In one of the embodiments, the multi-modal large model is used to analyze the detection results of the object behavior in the target image in the period and generate a detection report, which includes:

[0023] Based on the detection object and the object behavior, the multi-modal large model identifies whether the detection object has the object behavior in the target image, and if so, records the occurrence time of the object behavior.

[0024] Until the period for generating the detection report is met, the multi-modal large model analyzes and generates the detection report based on the recorded occurrence time, the detection object, and the object behavior.

[0025] In one of the embodiments, the server needs to detect multiple detection objects and multiple object behaviors sent by the mobile terminal in a unit of time, and one detection object is associated with at least one object behavior.

[0026] The multi-modal large model is used to analyze the detection results of the object behavior in the target image in the period and generate a detection report, which includes:

[0027] The detection object and its associated object behavior are taken as a detection combination, and for each detection combination, the multi-modal large model identifies whether the detection object of the detection combination has the object behavior of the detection combination in the detection image, and if so, records the occurrence time of the object behavior.

[0028] Until the period for generating the detection report is met, the occurrence time of the object behavior associated with each detection object is determined, and the multi-modal large model generates a detection report based on the detection object and the occurrence time of the associated object behavior.

[0029] In a second aspect, the present application also provides a home scene monitoring method supporting custom detection events, which is applied to a mobile terminal of a monitoring system, the monitoring system further comprising a server and an image acquisition device, the image acquisition device being used to acquire detection images in a home scene, and the method comprising:

[0030] A user-defined detection event is determined, and the detection event is sent to the server to enable the server to execute the above method based on the detection event and the detection images acquired by the image acquisition device, wherein the detection event includes at least one detection object and at least one object behavior related to the detection object.

[0031] In a third aspect, the present application further provides a monitoring system supporting custom detection events, comprising:

[0032] one or more image acquisition devices configured to acquire detection images in a home scene;

[0033] one or more mobile terminals configured to obtain user-defined detection events;

[0034] a cloud server configured to communicate with the image acquisition devices and the mobile terminals to perform the home scene monitoring method of the first aspect.

[0035] In a fourth aspect, the present application further provides an electronic device applied to a server end, comprising:

[0036] a memory for storing a computer program; and a processor for executing the computer program, the computer program being executed by the processor to implement the steps of the home scene monitoring method of the first aspect.

[0037] In the above-mentioned home scene monitoring method, system and electronic device supporting custom detection events, the monitoring method is applied to the server end of the monitoring system, and the server end is configured with a multi-modal large model; the monitoring system further comprises mobile terminals and image acquisition devices in communication with the server end, the mobile terminals being configured to obtain user-defined detection events, and the detection events comprising at least one detection object and at least one object behavior related to the detection object. Specifically, the server first acquires detection images acquired by the image acquisition devices, and then receives detection events sent by the mobile terminals. The detection events are set by users according to their specific needs, such as detecting whether a pet enters a specific area. This personalized setting method enables the monitoring method to meet the monitoring needs of different complex behaviors or different objects in a home scene. Using the multi-modal large model to determine whether the detection images contain the detection object and the object behavior related to the detection object in the detection events can realize corresponding intelligent detection based on user individualization or customization. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 Application environment diagram of a home scene monitoring method supporting custom detection events in an embodiment;

[0039] Figure 2 Flowchart of a home scene monitoring method supporting custom detection events in an embodiment;

[0040] Figure 3 Flowchart of determining whether a pet in a home has corresponding object behaviors in an embodiment;

[0041] Figure 4 This is a flowchart illustrating the process of determining whether a child in the home exhibits a corresponding object behavior in one embodiment.

[0042] Figure 5 This is a schematic diagram illustrating the determination of whether a child is on a windowsill in one embodiment;

[0043] Figure 6 This is a schematic diagram of the process for generating a detection report using a multimodal large model in one embodiment;

[0044] Figure 7 This is a schematic diagram of a process in one embodiment that uses a multimodal large model to record each detection combination and generate a detection report for each detection combination;

[0045] Figure 8 This is a schematic diagram illustrating the specific process of a home scene monitoring method that supports custom detection events in one embodiment;

[0046] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0048] like Figure 1 The monitoring system shown is used for monitoring home scenes. This home scene monitoring system is used to understand the situation in a specific area both locally and remotely, and can monitor the real-time situation of key areas around the residence. The monitoring system includes a terminal 101, a server 102, and an image acquisition device 103. The home monitoring method supporting customized detection events provided in this application embodiment is applied to the server 102. The terminal 101 can communicate with the server 102 via a network. Figure 1The data storage system in the server 102 can be integrated on the server 102, or placed on a cloud or other network server. The user sets the detection event through the terminal 101, and the terminal 101 sends the detection event set by the user to the server 102. The server 102 acquires the detection image collected by the image collection device 103, determines whether there is a detection object and an object behavior related to the detection object based on the multi-modal large model in the server, and then sends the detection result to the terminal 101. The terminal 101 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, wearable smart devices, etc. The server 102 can be implemented by a single server or a server cluster composed of multiple servers. The image collection device can be, but is not limited to, a fixed camera, a network camera, a wireless camera, a rotatable camera, a panoramic camera, etc.

[0049] In one embodiment, as shown in Figure 2 A home scene monitoring method supporting custom detection events is provided. The application is applied to the server end of a monitoring system, and the server end is configured with a multi-modal large model. The monitoring system further includes a mobile terminal and an image collection device in communication with the server end. The detection event includes at least one detection object and at least one object behavior related to the detection object. The method includes the following steps:

[0050] Step 201: Acquire the detection image collected by the image collection device;

[0051] Specifically, the server acquires the detection image collected by the image collection device, which can be a real-time image captured by a camera. It should be noted that the requirement for the camera in this embodiment is not high, and any camera that can provide stable shooting can meet the needs of the monitoring system.

[0052] Step 202: Receive the detection event sent by the mobile terminal, and determine whether the detection image contains a detection object using the multi-modal large model; and take the detection image containing the detection object as a target image;

[0053] Specifically, the server receives a detection event sent by the mobile terminal, and the detection event can refer to a detection object and at least one object behavior related to the detection object. For example, a puppy at home on the sofa, where the puppy at home is the detection object, and sitting on the sofa is the object behavior related to it; a cat drinking water and eating cat food, where the cat is the detection object, and drinking water and eating cat food are object behaviors corresponding to it. It should be noted that the detection object in the detection event can be transmitted to the server by the mobile terminal in the form of text description, image or other ways. If the detection object in the detection event is represented in the form of an image, after the server obtains the detection image collected by the image collection device, it calculates the image of the detection object in the detection event and each frame of the detection image collected by the image collection device using a multi-modal large model to determine whether the detection image collected by the image collection device contains the image of the detection object in the detection event, and takes the multiple frames of the detection image containing the detection object as the target image. For example, the detection object in the detection event is a pet at home. If a frame of the detection image collected by the image collection device contains the image of the pet at home, then the frame of the detection image collected by the image collection device can be used as the target image; if a frame of the detection image collected by the image collection device does not contain the image of the pet at home, then the detection image collected at this time cannot be used as the target image.

[0054] It should be noted that the multi-modal large model is an artificial intelligence technology that can process and understand multiple different types of data input, such as text, images, audio and video, etc. This model learns how to recognize and match different types of data through deep learning and large-scale data training. In identifying whether a specific detection object exists in the target image, if the detection event contains text descriptions, the multi-modal large model will use a text encoder to process these descriptions and understand their semantic content; if it directly contains image data of the detection object, the model will extract image features through a visual encoder. Then, the model uses modal fusion techniques, such as a visual-text alignment layer, to combine image features with text features to achieve cross-modal reasoning. In this way, the model can perform feature extraction on a large number of images collected by the image collection device after the user inputs the detection event, and compare it with the detection event to accurately determine whether the detection object exists in the detection image.

[0055] Step 203: Determine whether the object behavior exists in the target image using a multi-modal large model.

[0056] Specifically, the target image is analyzed by using a multi-modal large model to determine whether there is an object behavior related to the detection object in the target image. For example, if the object behavior related to the detection object is that a small dog at home is on the sofa, if the small dog does not have the behavior of being on the sofa in the target image, then the detection object does not perform the object behavior at this time; only when the target image and the object behavior exist at the same time, such as the small dog appearing on the sofa in the target image, can it be determined that the detection object and the object behavior related to the detection object exist.

[0057] The home scene monitoring method supporting custom detection events provided by the above embodiment, the server first acquires the detection image collected by the image collection device, and then the server receives the detection event sent by the mobile terminal. For the image collection device, a normal camera can be used to complete the shooting required by the system, so the cost demand for the image collection device will not be high. The detection event is set by the user according to his specific needs, such as detecting whether a pet enters a specific area. This personalized setting method makes the monitoring method more in line with the actual needs of the user. Further, the multi-modal large model is used to determine whether the detection image contains the detection object, that is, the image collected by the image collection device contains the detection object in the detection event, and the detection image containing the detection object is used as the target image. The multi-modal large model is used to determine whether the detection image contains the object behavior of the detection object related to the detection object in the detection event. Further, in this embodiment, this monitoring method provides users with a more reliable monitoring experience through personalized user settings and intelligent detection, without excessive requirements for image collection devices and additional hardware costs.

[0058] In one embodiment, the detection object is an object whose position coordinates in the home scene can change; the detection image of the image collection device is acquired, including the following steps:

[0059] In response to the event instruction of the image collection device detecting the object movement event, a screen capture instruction is sent to the image collection device to make the image collection device save the detection image in response to the screen capture instruction and upload the detection image to the server; wherein the image collection device starts the movement detection function in response to the start instruction sent by the mobile terminal, and sends an event instruction to the server in the case of detecting the object movement event.

[0060] Specifically, when the image acquisition device detects that an object moves, the image acquisition device sends an event instruction to the server, and when the server receives the instruction, the server sends a picture capturing instruction to the image acquisition device. At this time, the image acquisition device starts to capture pictures after receiving the instruction, saves the pictures and uploads them to the server. It should be noted that the user can send an instruction to the image acquisition device through the mobile terminal to start the movement detection function. At this time, the image acquisition device will send an event instruction to the server as soon as it detects that an object moves.

[0061] In this embodiment, when the image acquisition device detects that an object moves, the image acquisition device sends an event instruction to the server, and when the server receives the instruction, the server sends a picture capturing instruction to the image acquisition device. At this time, the image acquisition device starts to capture pictures after receiving the instruction, saves the pictures and uploads them to the server. It should be noted that the user can send an instruction to the image acquisition device through the mobile terminal to start the movement detection function. At this time, the image acquisition device will send an event instruction to the server as soon as it detects that an object moves.

[0062] In one embodiment, as shown in Figure 3 The detection object is a pet in a home scene, and the object behavior is that the pet moves to a specific object in the home scene. A multi-modal large model is used to determine whether the target image contains the corresponding object behavior, including the following steps:

[0063] Step 301: Determine the bounding box of the pet and the bounding box of the specific object in the target image based on the multi-modal model, wherein the multi-modal model includes a pet recognition model.

[0064] Specifically, the server uses a pet recognition model in the multi-modal large model, wherein the pet recognition model can recognize multiple pets and generate a bounding box for each pet. The server can also use the multi-modal large model to obtain the bounding box of the specific object. The bounding box is usually determined by the coordinates of the upper left corner and the lower right corner of a rectangle. For example, the object behavior of the target image is that a kitten runs onto a dining table, wherein the kitten is a pet and the dining table is a specific object. The server can use a specific kitten recognition model in the pet recognition model of the multi-modal large model to recognize the kitten and generate a bounding box for it, and use the multi-modal large model to generate a bounding box for the dining table.

[0065] Step 302: Obtain the coordinates of the two bounding boxes, and determine the intersection region between the two bounding boxes based on the coordinates of the two bounding boxes.

[0066] Specifically, the server calculates, by using the multi-modal large model, the recognition boxes of the pet and the specific object, and the coordinates of the upper left corner and the lower right corner corresponding to the two recognition boxes. Based on the coordinates corresponding to the two recognition boxes, the intersection part between the two recognition boxes can be obtained, and the intersection part between the two recognition boxes is the intersection region between the two recognition boxes.

[0067] Step 303: If the area of the intersection region is greater than a set threshold, the object behavior exists in the target image.

[0068] In the above example, the server calculates, by using the multi-modal large model, the area of the intersection region between the cat recognition box and the dining table recognition box based on the coordinates of the intersection region. After obtaining the area of the intersection region, the area is compared with the threshold set by the user. If the area of the intersection region between the cat recognition box and the dining table recognition box is greater than the set threshold, it indicates that the cat and the dining table have an intersection relationship, i.e., the cat and the dining table have an object behavior. If the area of the intersection region between the cat recognition box and the dining table recognition box is less than the set threshold, it cannot be determined that the cat and the dining table have an intersection relationship, i.e., it cannot be determined that the cat and the dining table have an object behavior.

[0069] In this embodiment, the server can accurately identify the recognition boxes of the pet and the specific object by using the multi-modal large model, thereby improving the detection accuracy. Meanwhile, the server can obtain the coordinates of the two recognition boxes by using the multi-modal large model, and calculate the area of the intersection region between the two recognition boxes. By setting the area threshold of the intersection region by the user, the server can efficiently determine whether the pet and the specific object have an intersection, thereby reducing the possibility of false positives and false negatives.

[0070] In one embodiment, as shown in FIG. 4, the detected object is a child in a home scene, and the object behavior is that the child moves to a specific object, and the specific object is an object with fixed position coordinates in the home scene. The multi-modal large model is used to determine whether the corresponding object behavior exists in the target image, including the following steps: Figure 4

[0071] Step 401: Determine the recognition box of the child and the recognition box of the specific object in the target image based on the multi-modal large model, wherein the child recognition box is taken as a first recognition box, and the specific object recognition box is taken as a second recognition box;

[0072] ​Specifically, in a home scenario, parents need to monitor the interaction between children and certain specific objects to ensure the safety of children. For example, a child stands on a windowsill for a target image related to the object behavior. The server generates a recognition box for the child in the target image using a child recognition model in the multi-modal large model, i.e., a first recognition box. Similarly, the server generates a recognition box for the specific object (here, the windowsill) using a specific object recognition model in the multi-modal large model, i.e., a second recognition box. The resulting recognition boxes for the child and the specific object generally include the coordinates of the upper left corner and the lower right corner of a rectangle.

[0073] Step 402: If there is an intersection between the first recognition box and the second recognition box, determine whether the center of the first recognition box is above the center of the second recognition box.

[0074] Specifically, the server uses the multi-modal large model to calculate the first recognition box for the child and the second recognition box for the specific object, which can be the windowsill. Based on the coordinates of the two recognition boxes, the intersection range between the two recognition boxes is calculated to determine whether their positions intersect. Only when the two recognition boxes intersect can it be determined that the child and the windowsill have a positional intersection relationship. In this example, the child stands on the windowsill for the target image related to the object behavior. The emphasized behavior is that the child is on the windowsill, so it is necessary to determine whether the center of the child is above the center of the windowsill, i.e., whether the center of the first recognition box is above the center of the second recognition box. The center coordinates of the first recognition box and the second recognition box can be calculated by the formula Here, x min , y max are the coordinates of the upper left corner of the recognition box, x max , y min are the coordinates of the lower right corner of the recognition box, and (x c , y c ) is the center point of the recognition box. Only when the center point coordinates of the first recognition box are above the center point coordinates of the second recognition box can it be determined that there is an object behavior of the child on the windowsill.

[0075] Step 403: If so, determine whether the length-width ratio of the first recognition box is less than a predetermined ratio.

[0076] Specifically, according to the above steps, the purpose of determining the length-width ratio of the first recognition box is to determine the state of the child's action. When the child is on the windowsill, standing posture is generally more likely to be dangerous than lying posture. Therefore, the server uses the multi-modal large model to calculate the coordinates of the first recognition box to obtain the length-width ratio of the first recognition box, where the length-width ratio of the first recognition box can be calculated using where x min , y max are the coordinates of the upper left corner of the first recognition box, x max , ymin is the right bottom coordinate of the first recognition box. The length-width preset ratio here can be 1, x max -x min represents length, y max -y min represents width. When the length-width ratio of the first recognition box is less than the preset ratio, the width of the first recognition box of the child is greater than the length, i.e., the child can be in a standing posture, and the possibility of dangerous behavior on the windowsill is great; if the length-width ratio of the first recognition box is greater than the preset ratio, the width of the first recognition box of the child is less than the length, i.e., the child can be in a prone action, and the possibility of dangerous behavior on the windowsill is small.

[0077] Step 404: If yes, it is determined that the target image exists an object behavior.

[0078] Specifically, by the above steps, it is determined that the target image exists an object behavior. That is, in the above example, if the center of the first recognition box representing the child is located above the center of the second recognition box representing the windowsill, and the length-width ratio of the first recognition box of the child is less than the preset ratio, it is indicated that the target image exists an object behavior that the child stands on the windowsill, and the server sends a notification to the mobile terminal.

[0079] It should be noted that, as Figure 5 shown, for example, the detection event of the child standing on the windowsill, if the child and the windowsill exist a positional intersection relationship, and the center of the child is located above the center of the windowsill, it can be judged that the child stands on the windowsill. When the recognition boxes of the two appear intersection and the intersection area is greater than a set threshold, it is indicated that the two exist a positional intersection relationship. Secondly, the up-down relationship of the center points of the two can be calculated according to the coordinates of the recognition boxes, which can be calculated by the formula . Wherein x min , y max are the left top coordinates of the recognition box, x max , y min are the right bottom coordinates of the recognition box, (x c , y c ) is the center point of the recognition box. When the recognition boxes of the child and the windowsill exist intersection and the center point coordinate of the child is above the center point coordinate of the windowsill, the length-width preset ratio can be 1, x max -x min represents length, y max -y min represents width. When the length-width ratio of the first recognition box is less than the preset ratio, it is indicated that the child stands on the windowsill, and the server will send a notification to the mobile terminal.

[0080] In this embodiment, the server determines the bounding box of the child and the window sill by using the multi-modal large model, and judges whether the center of the child bounding box is above the center of the window sill bounding box, and whether the length-width ratio is less than a preset ratio, to determine whether the child stands on the window sill. When each condition is met, it is determined that the object behavior exists, effectively reducing the misjudgment. If the child indeed has an object behavior with a specific object, the server can immediately send a notification to the mobile terminal, so that the user can timely discover the potential safety hazard that the child may face.

[0081] In one embodiment, obtaining the detection event sent by the mobile terminal comprises: in response to the detection request sent by the mobile terminal, parsing the detection request to determine the detection event sent by the mobile terminal; wherein the detection event includes not only the detection object and the object behavior, but also the period for generating the detection report; the method further comprises:

[0082] If the detection image contains the detection object, the server sends a notification to the mobile terminal, and when the period for generating the detection report is reached, the multi-modal large model is used to analyze the detection results of the object behavior in the target image in the period and generate a detection report, which is sent to the mobile terminal.

[0083] Specifically, the user can send a detection event through the mobile terminal, which contains the detection object, the object behavior related to the detection object, and the period for generating the detection report. The server receives the detection event from the mobile terminal and parses it to determine the specific content and requirements of the detection event, i.e. the detection object and the object behavior and the period for generating the detection report. The server uses the multi-modal large model to judge whether each frame of detection image obtained by the image acquisition device contains the detection object, and if it contains the detection object, the corresponding frame of detection image containing the detection object is taken as the target image and a notification is sent to the mobile terminal. The server sets a monitoring period according to the parsing of the detection event content, and starts monitoring at the beginning of each period. In each period, the server continuously obtains the detection images transmitted by the image acquisition device, and uses the multi-modal large model to judge which frames of detection images obtained by the image acquisition device contain the detection object and the object behavior related to it and record the time when the object behavior occurs. At the end of each monitoring period, the server analyzes the detection results in the period by using the multi-modal large model to generate a detailed detection report. The server sends the detection report to the mobile terminal of the user, and the user can understand the occurrence of the detection object and the related object behavior in the detection event defined by the user within the period.

[0084] In this embodiment, the user can receive the notification sent by the server in real time through the mobile terminal, and the user can periodically receive the periodic detailed report of the custom detection event. This helps the user to better understand and monitor the security situation in the family. The server can automatically collect and analyze data, reduce the user's analysis of a large number of detection results, and improve the monitoring efficiency. At the same time, the user can analyze the detection results in the analysis period, understand the behavior habits of the detection objects such as children or pets, and provide effective data support for family education and safety prevention.

[0085] In one embodiment, as shown in FIG. 6, the detection results of the target image in the period about the object behavior are analyzed and a report is generated by using a multi-modal large model, including the following steps: Figure 6

[0086] Step 601: Based on the detection object and the object behavior, whether the detection object in the target image exists the object behavior is identified by using a multi-modal large model, and if so, the occurrence time of the object behavior is recorded.

[0087] Specifically, the user sets the monitoring parameters through the mobile terminal, including the detection object, the object behavior, and the monitoring period. For example, whether the child in the family approaches the plug-in board from 8:00 to 10:00 every day, wherein the detection object is the child, the object behavior is approaching the plug-in board, and the monitoring period is from 8:00 to 10:00 every day. In the set monitoring period, the image acquisition device continuously captures the images in the monitoring area. After the server receives the images uploaded by the image acquisition device, the target image is analyzed and identified by using a multi-modal large model, to determine whether the detection object in the target image exists the corresponding object behavior, i.e., whether the image of the child exists the behavior of approaching the plug-in board. If the multi-modal large model determines that the corresponding object behavior exists, the server will record the occurrence time of the object behavior. That is, if the child approaches the plug-in board at 9:25 in the morning, the time will be recorded by the server.

[0088] Step 602: Until the period for generating the detection report is met, the multi-modal large model analyzes and generates the detection report based on the recorded occurrence time, the detection object, and the object behavior.

[0089] Specifically, until the period for generating the detection report is met, i.e., after the monitoring period starts and ends, the multi-modal large model analyzes the collected data, including the occurrence time and frequency of the object behavior. The server can generate a detection report according to the analysis result, and the report contains the most detailed statistical data of the object behavior, such as the time, the number of times, the time period of the most frequent approach to the plug-in board, etc.

[0090] ​In this embodiment, the monitoring system can accurately record the time when the detection object and its corresponding object behavior occur, providing detailed data for subsequent generation of periodic reports. Through the generated periodic detection report, the user can regularly understand the behavior pattern of the detection object, so that the user does not need to continuously monitor for a long time, improving the efficiency of monitoring. The generated detection report of the multi-modal large model can contain key information such as the frequency time period distribution of the behavior occurrence, helping the user better understand the behavior habits of the detection object. The user can customize the monitoring period and behavior according to his own needs, making the system more convenient and meeting the needs of different groups of people.

[0091] In one embodiment, as shown in FIG. 7, in a unit time, the server needs to detect multiple detection objects and multiple object behaviors sent by the mobile terminal, and one detection object manages at least one object behavior; using a multi-modal large model, the detection results of the object behaviors in the target image in the period are analyzed and a detection report is generated, including the following steps: Figure 7

[0092] Step 701: Take one object behavior expected to be associated with a detection object as a detection combination, for each detection combination, use a multi-modal large model to identify whether the detection object in the detection combination occurs the object behavior in the detection combination in the detection image, if yes, record the time when the object behavior occurs;

[0093] Specifically, in a unit time, the server needs to detect multiple detection objects and multiple object behaviors sent by the mobile terminal, and one detection object manages at least one object behavior. Each detection object and the object behavior related to it form a detection combination. The user can send, for example, a kitten eating cat food, a kitten eating on the dining table, a child climbing on the sofa, an old person coming home, etc. Among them, the kitten can correspond to two object behaviors of drinking water and eating on the dining table, the child corresponds to one object behavior of climbing on the sofa, and the old person corresponds to one object behavior of coming home. These objects and object behaviors form a detection combination, where the kitten drinking water and the kitten eating on the dining table are two detection combinations. The server uses a multi-modal large model to determine whether the detection object in the detection combination occurs the object behavior related to the detection object in the detection combination in the target image. If yes, record the time when the object behavior in the detection combination occurs.

[0094] Step 702: Until the period for generating a detection report is met, for each detection object, determine the occurrence time of the object behavior associated with the detection object, and the multi-modal large model generates a detection report based on the detection object and the occurrence time of the associated object behavior.

[0095] ​Specifically, until the end of the detection cycle time, the server records the occurrence time of the object behavior of each detection object in each detection combination and its related object behavior by using the multi-modal large model. Secondly, the server generates a detection report based on the detection object and the occurrence time of the associated object behavior by using the multi-modal large model. That is, according to the example in step 701, the multi-modal large model generates a detection report for the object behavior occurrence time recorded in the detection combination of cat eating cat food, cat on the table, child climbing on the sofa, and old people coming home. It should be noted that the monitoring period of each detection combination can be different, for example, the detection objects of cat eating cat food, cat on the table, and child climbing on the sofa can be the monitoring period from 9:00 am to 6:00 pm, and the old people coming home can be the monitoring period from 9:00 am to 11:00 am when the old people go for a walk. For each different detection combination, the server can record the corresponding behavior occurrence time and generate a detection cycle report corresponding to each detection combination.

[0096] In this embodiment, by grouping different detection objects and their corresponding at least one object behavior into a detection combination, the server can simultaneously process multiple detection objects and behaviors, so the server can more efficiently utilize computing resources, reduce repeated detection and analysis processes, and comprehensively and in detail monitor multiple behaviors of each detection object. For each detection combination, the server can accurately record the occurrence time of the object behavior, provide accurate data support for subsequent analysis, and generate a detection report that can serve as data support to help users make more reasonable response decisions based on actual object behavior.

[0097] In one embodiment, a monitoring method for a home scene supporting custom detection events is applied to a mobile terminal of a monitoring system, the monitoring system further comprising a server and an image acquisition device, and the method comprises:

[0098] determining a user-defined detection event, and sending the detection event to the server to enable the server to execute the above-mentioned various embodiments based on the detection event and the detection image acquired by the image acquisition device, wherein the detection event comprises at least one detection object and at least one object behavior related to the detection object.

[0099] Specifically, the user sends a detection event customized by the user to the server through the mobile terminal, the detection event including at least one detection object and at least one object behavior related to the detection object and a monitoring period. After receiving the detection event, the server analyzes and stores it in the database for subsequent processing. After the server receives the image uploaded by the image collection device, the target image is recognized by using the multi-modal large model analysis to determine whether the object behavior of the detection object exists in the target image. If the user-defined detection event is detected, the server records and sends a notification to the mobile terminal. The server can also generate a periodic report based on the recorded detection event data after the set monitoring period ends. The report is sent to the user through the mobile terminal, which includes the frequency of the object behavior occurring and the time period of the behavior occurring within the period.

[0100] In this embodiment, through the home scene monitoring method, the user can customize the detection event according to his own needs and family situation, making the monitoring system more personalized, and also receiving timely notification of the detection event at home, improving the grasp of the situation at home. Through the periodic report, the user can understand the behavior patterns of family members and pets, providing data support for family education and pet management. At the same time, the high accuracy of the multi-modal large model also reduces the possibility of false positive notifications.

[0101] In one embodiment, as shown in Figure 8 The home scene monitoring method supporting custom detection events includes the following specific processes:

[0102] Step 801: The mobile terminal sends a mobile detection start instruction to the image collection device;

[0103] Step 802: The mobile terminal sends the custom detection object and object behavior to the server;

[0104] Step 803: The mobile terminal sends the custom detection report generation period to the server;

[0105] Step 804: The server responds to the event instruction of the image collection device detecting object movement events;

[0106] Step 805: The server sends a screenshot instruction to the image collection device;

[0107] Step 806: The image collection device saves the detection image and uploads the detection image to the server;

[0108] Step 807: The server analyzes the detection object set by the user and the image uploaded by the camera based on the multi-modal large model;

[0109] Step 808: The server sends the detection result containing the detection object set by the user to the mobile terminal;

[0110] Step 809: The user views the daily detection results sent by the server through the mobile terminal;

[0111] Step 810: When the reporting period set by the user is reached, the detection results of the recent period are classified and arranged based on the multi-modal large model, and suggestions are prompted.

[0112] Step 811: The user views the periodic detection report sent by the server through the mobile terminal.

[0113] In this embodiment, by integrating image acquisition devices, mobile terminals and cloud servers, an efficient and customizable monitoring solution is achieved. Users can set detection events according to their own needs, and the system automatically performs monitoring and analysis tasks, notifies users of important events in a timely manner, and provides periodic reports to enhance home security and convenience.

[0114] Based on the same inventive concept, the embodiments of the present application also provide a monitoring system for implementing the above-mentioned home scene monitoring method supporting custom detection events. Specifically, a monitoring system supporting custom detection events includes:

[0115] One or more image acquisition devices configured to acquire detection images in a home scene;

[0116] One or more mobile terminals configured to obtain user-defined detection events;

[0117] A cloud server configured to communicate with the image acquisition device and the mobile terminal to perform the steps of the above-mentioned embodiments of the home scene monitoring method supporting custom detection events.

[0118] Specifically, one or more cameras are installed in the home environment as image acquisition devices, which are configured to acquire images in the home scene in real time. The user defines detection events on the mobile terminal, such as monitoring whether the elderly in the home have fallen or whether pets have entered a certain area, and sends these events to the cloud server. The user turns on the mobile detection function of the camera through the mobile terminal to the image acquisition device, and sends an event instruction to the server after detecting the movement event of the object. When the server receives the instruction, it sends a picture grabbing instruction to the image acquisition device, at which time the image acquisition device continuously acquires images and continuously transmits the images to the cloud server. The cloud server receives the image data and analyzes the images using a multi-modal large model to identify the detection objects and behaviors defined by the user. When the server detects a behavior that matches the user-defined detection event, it sends a record and notifies the user's mobile terminal. The cloud server can also generate a report containing detection event analysis periodically and provide it to the user through the mobile terminal so that the user can understand the long-term trend of the monitoring event.

[0119] In one embodiment, an electronic device is also provided, which is applied to a server side, and the electronic device can also be a server. Specifically, the electronic device comprises:

[0120] a memory for storing computer programs;

[0121] a processor for executing the computer programs, and when the computer programs are continuously executed by the processor, each step in the above embodiments is implemented.

[0122] The internal structure of the electronic device can be as shown in Figure 9 The electronic device comprises a memory and a processor, the memory is used to store computer programs, and the processor is used to execute these programs. In the memory, a series of computer programs are stored, which contain codes for implementing the monitoring system, such as image acquisition, user-defined event processing, multi-modal large model analysis, etc. The processor loads and executes these computer programs from the memory, thereby implementing each step. When the processor executes the image acquisition program, it will control the image acquisition device to acquire images in the home scene. When the user-defined event processing program is executed, the processor will receive user-defined detection events from the mobile terminal and configure monitoring parameters according to these events. When the multi-modal large model analysis program is executed, the processor will analyze the acquired image data to identify detection objects and behaviors. In addition, the processor will also generate a detection report according to the user-set monitoring period and send the report to the terminal to feedback to the user, so that the user can understand the general trend in the monitoring period.

[0123] In this embodiment, the electronic device realizes an efficient and automated monitoring system through the cooperation of its memory and processor. It can process a large amount of image data, identify and respond to user-defined detection events, and provide timely notifications and periodic reports, thereby improving the intelligent level of home monitoring and enhancing the user's monitoring experience. At the same time, this device is flexible and can perform different monitoring tasks according to the user's specific needs.

[0124] Those skilled in the art can understand that, Figure 9 the structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0125] The above embodiments only express several implementation ways of the present application, and the description is specific and detailed, but it should not be understood as a limitation to the patent scope of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A home scene monitoring method supporting a custom detection event, characterized by, The application is applied to a server end of a monitoring system, the server end is configured with a multi-modal large model; the monitoring system further comprises a mobile terminal and an image acquisition device in communication with the server end, the mobile terminal is configured to obtain a detection event customized by a user, the detection event comprises at least one detection object and at least one object behavior related to the detection object; the method comprises: obtaining a detection image collected by the image acquisition device; receiving the detection event sent by the mobile terminal, and determining whether the detection object is contained in the detection image by using the multi-modal large model; and taking the detection image containing the detection object as a target image; determining whether the corresponding object behavior exists in the target image by using the multi-modal large model, and notifying the mobile terminal if the corresponding object behavior exists in the target image; if the detection object is a child in a home scene, and the object behavior is that the child moves to a specific object, and the specific object is an object with fixed position coordinates in the home scene; determining whether the corresponding object behavior exists in the target image by using the multi-modal large model, comprising: determining the recognition box of the child and the recognition box of the specific object in the target image based on the multi-modal large model, wherein the recognition box of the child is taken as a first recognition box, and the recognition box of the specific object is taken as a second recognition box; if there is an intersection area between the first recognition box and the second recognition box, it is determined whether the center of the first recognition box is located above the center of the second recognition box; if yes, it is determined whether the length-width ratio of the first recognition box is less than a preset ratio; if yes, it is determined that the object behavior exists in the target image.

2. The home scene monitoring method according to claim 1, characterized by, The detection object is an object with changeable position coordinates in the home scene; obtaining the detection image collected by the image acquisition device comprises: in response to an event instruction of detecting an object movement event of the image acquisition device, sending a picture grabbing instruction to the image acquisition device, so that the image acquisition device saves the detection image and uploads the detection image to the server in response to the picture grabbing instruction; wherein the image acquisition device starts the movement detection function in response to the start instruction sent by the mobile terminal, and sends the event instruction to the server end when detecting the object movement event.

3. The home scene monitoring method according to claim 1, characterized by, The detection object is a pet in the home scene, and the object behavior is that the pet moves to a specific object in the home scene; determining whether the corresponding object behavior exists in the target image by using the multi-modal large model comprises: determining the recognition box of the pet and the recognition box of the specific object in the target image based on the multi-modal model, the multi-modal model comprising a pet recognition model; obtaining the coordinates of the two recognition boxes, and determining the intersection area between the two recognition boxes based on the coordinates of the two recognition boxes; if the area of the intersection area is greater than a set threshold, the object behavior exists in the target image.

4. The home scene monitoring method according to any one of claims 1-3, characterized in, The detection event sent by the mobile terminal comprises: in response to the detection request sent by the mobile terminal, parsing the detection request to determine the detection event sent by the mobile terminal; wherein, in addition to the detection object and the object behavior, the detection event also includes a period for generating a detection report; The method further comprises: If the detection image contains a detection object, the server sends a notification to the mobile terminal, and when the period for generating a detection report is reached, the multi-modal large model is used to analyze the detection results of the object behavior in the target image within the period and generate a detection report, which is sent to the mobile terminal.

5. The home scene monitoring method according to claim 4, characterized in that, Using the multi-modal large model to analyze the detection results of the object behavior in the target image within the period and generate a detection report comprises: Based on the detection object and the object behavior, the multi-modal large model is used to identify whether the detection object exists in the target image, and if so, the occurrence time of the object behavior is recorded; Until the period for generating a detection report is met, the multi-modal large model analyzes and generates a detection report based on the recorded occurrence time, the detection object, and the object behavior.

6. The home scene monitoring method according to claim 5, characterized in that, In a unit of time, the server needs to detect multiple detection objects and multiple object behaviors sent by the mobile terminal, and one detection object is associated with at least one object behavior; Using the multi-modal large model to analyze the detection results of the object behavior in the target image within the period and generate a detection report comprises: The detection object and its associated object behavior are regarded as a detection combination, and for each detection combination, the multi-modal large model is used to identify whether the detection object of the detection combination occurs in the detection image, and if so, the occurrence time of the object behavior is recorded; Until the period for generating a detection report is met, for each detection object, the occurrence time of the object behavior associated with the detection object is determined, and the multi-modal large model generates a detection report based on the detection object and the occurrence time of the associated object behavior.

7. A home scene monitoring method supporting a custom detection event, characterized by, The mobile terminal applied to the monitoring system, the monitoring system further comprises a server and an image acquisition device, the image acquisition device is used to acquire detection images in a home scene, and the method comprises: Determine the detection event customized by the user, and send the detection event to the server to enable the server to execute the method according to any one of claims 1 to 6 based on the detection event and the detection image acquired by the image acquisition device, wherein the detection event comprises at least one detection object and at least one object behavior related to the detection object.

8. A monitoring system supporting custom detection events, characterized by The monitoring system comprises: One or more image acquisition devices configured to acquire detection images in a home scene; One or more mobile terminals configured to acquire user-defined detection events; A cloud server configured to communicate with the image acquisition device and the mobile terminal to execute the steps of the method according to any one of claims 1 to 6.

9. An electronic device, applied to a server side, characterized in that, Comprise: Memory for storing computer programs; a processor for executing said computer program, said computer program implementing the steps of the method according to any one of claims 1 to 6 when executed by the processor.

Images