Home scene monitoring method and monitoring system supporting custom detection event

Abstract

The invention relates to a home scene monitoring method supporting a user-defined detection event, the method is applied to a server side of a monitoring system, and the server side is configured with a multi-modal large model; the monitoring system further comprises a mobile terminal communicating with the server side and an image acquisition device, the mobile terminal is configured to acquire a user-defined detection event, and the detection event comprises at least one detection object and at least one object behavior related to the detection object; the method comprises the following steps: acquiring a detection image acquired by image acquisition equipment; receiving a detection event sent by the mobile terminal, and determining whether the detection image contains a detection object or not by using a multi-modal large model; taking the detection image containing the detection object as a target image; and determining whether a corresponding object behavior exists in the target image by using a multi-modal large model, and if so, notifying the mobile terminal. According to the method, intelligent detection can be realized based on personalized setting of the user, and flexible and efficient monitoring experience is provided for the user.

Description

Technical Field

[0001] The present application relates to the field of smart home technology, and in particular to a home scene monitoring method, monitoring system and electronic equipment that support customized detection events. Background Art

[0002] With the development of smart home technology, more and more users install cameras at home and connect and view them through mobile phones for security or communication reasons. Since users cannot always check the situation in the camera, they are likely to miss many emergencies.

[0003] In the field of smart home, cameras are usually installed in the interior of residents' homes, such as living rooms and rooms, to better monitor the situation at home. However, the ability to detect the specific behavior of a single object is very limited. Usually, it can only detect simple content such as whether a person moves, a vehicle appears, or a pet appears. Moreover, most cameras on the market can only use the monitoring settings set at the factory, which cannot meet the user's monitoring needs for different complex behaviors or different objects in the home scene. Summary of the invention

[0004] Based on this, it is necessary to provide a home scene monitoring method, monitoring system and electronic device that support customized detection events in response to the above technical problems.

[0005] In a first aspect, the present application provides a home scene monitoring method supporting customized detection events, which is applied to a server side of a monitoring system, and the server side is configured with a multimodal large model; the monitoring system also includes a mobile terminal and an image acquisition device communicating with the server side, and the mobile terminal is configured to obtain a user-defined detection event, and the detection event includes at least one detection object and at least one object behavior related to the detection object; the home scene monitoring method includes:

[0006] Acquire a detection image acquired by an image acquisition device;

[0007] Receive a detection event sent by a mobile terminal, and use a multimodal large model to determine whether the detection image contains a detection object; and use the detection image containing the detection object as a target image;

[0008] Using the multimodal large model, determine whether there is a corresponding object behavior in the target image, and if so, notify the mobile terminal.

[0009] In one embodiment, the detection object is an object in a home whose position coordinates can change; the detection image acquired by the image device includes:

[0010] In response to an event instruction of an image acquisition device detecting an object movement event, a capture instruction is sent to the image acquisition device, so that the image acquisition device saves the detection image in response to the capture instruction and uploads the detection image to the server; wherein the image acquisition device turns on the motion detection function in response to the start instruction sent by the mobile terminal, and sends an event instruction to the server when an object movement event is detected.

[0011] In one embodiment, the detection object is a pet in the home, and the object behavior is the pet moving to a specific object in the home scene; using a multimodal large model, determining whether the target image has a corresponding object behavior includes:

[0012] Determining a recognition frame of a pet and a recognition frame of a specific object in a target image based on a multimodal model, wherein the multimodal model includes a pet recognition model;

[0013] Obtaining coordinates of the two recognition frames, and determining an intersection area between the two recognition frames based on the coordinates of the two recognition frames;

[0014] If the area of ​​the intersection region is greater than the set threshold, there is object behavior in the target image.

[0015] In one embodiment, the detection object is a child in a home scene, the object behavior is the child moving to a specific object, and the specific object is an object with fixed position coordinates in the home scene; using a multimodal large model, determining whether the target image has a corresponding object behavior includes:

[0016] Determine a recognition frame of a child and a recognition frame of a specific object in the target image based on the multimodal large model, wherein the recognition frame of the child is used as a first recognition frame, and the recognition frame of the specific object is defined as a second recognition frame;

[0017] If there is an intersection area between the first identification frame and the second identification frame, determining whether the center of the first identification frame is located above the center of the second identification frame;

[0018] If so, determining whether the aspect ratio of the first identification frame is less than a preset ratio;

[0019] If so, it is determined that object behavior exists in the target image.

[0020] In one embodiment, obtaining the detection event sent by the mobile terminal includes: in response to the detection request sent by the mobile terminal, parsing the detection request, and determining the detection event sent by the mobile terminal; wherein the detection event includes not only the detection object and the object behavior, but also the cycle of generating the detection report; the home scene monitoring method also includes:

[0021] If the detection image contains the detection object, the server sends a notification to the mobile terminal. When the cycle for generating the detection report is reached, the server uses a multimodal large model to analyze the detection results of the object behavior in the target image within the cycle and generate a detection report, which is sent to the mobile terminal.

[0022] In one embodiment, a multimodal large model is used to analyze the detection results of the object behavior in the target image within the period and generate a detection report, including:

[0023] Based on the detection object and the object behavior, a multimodal large model is used to identify whether the detection object in the target image has the object behavior. If so, the occurrence time of the object behavior is recorded;

[0024] Until the cycle for generating a test report is met, the multimodal large model analyzes and generates a test report based on the recorded occurrence time, test object, and object behavior.

[0025] In one of the embodiments, the server needs to detect multiple detection objects and multiple object behaviors sent by the mobile terminal within a unit time, and one detection object is associated with at least one object behavior;

[0026] Using a large multimodal model, the detection results of the object behavior in the target image within the cycle are analyzed and a detection report is generated, including:

[0027] The detection object and an object behavior associated with it are regarded as a detection combination. For each detection combination, the multimodal large model is used to identify whether the detection object of the detection combination has the object behavior of the detection combination in the detection image. If so, the occurrence time of the object behavior is recorded.

[0028] Until the cycle for generating a test report is met, for each test object, the occurrence time of the object behavior associated with the test object is determined, and the multimodal large model generates a test report based on the occurrence time of the test object and the associated object behavior.

[0029] In a second aspect, the present application further provides a home scene monitoring method supporting custom detection events, the home scene monitoring method is applied to a mobile terminal of a monitoring system, the monitoring system further includes a server and an image acquisition device, the image acquisition device is used to collect detection images in the home scene, the method includes:

[0030] Determine a user-defined detection event and send the detection event to the server side so that the server side can execute the above method based on the detection event and the detection image captured by the image acquisition device, wherein the detection event includes at least one detection object and at least one object behavior related to the detection object.

[0031] In a third aspect, the present application also provides a monitoring system supporting custom detection events, the monitoring system comprising:

[0032] One or more image acquisition devices configured to acquire detection images in a home scene;

[0033] One or more mobile terminals, configured to obtain user-defined detection events;

[0034] The cloud server is configured to communicate with the image acquisition device and the mobile terminal to execute the home scene monitoring method of the first aspect mentioned above.

[0035] In a fourth aspect, the present application further provides an electronic device, which is applied to a server side, and the electronic device includes:

[0036] The memory is used to store computer programs; the processor is used to execute the computer programs, and when the computer program is executed by the processor, the steps of the home scene monitoring method of the first aspect are implemented.

[0037] In the above-mentioned home scene monitoring method, system and electronic device that support customized detection events, the monitoring method is applied to the server side of the monitoring system, and the server side is configured with a multimodal large model; the monitoring system also includes a mobile terminal and an image acquisition device that communicate with the server side, and the mobile terminal is configured to obtain user-defined detection events, and the detection event includes at least one detection object and at least one object behavior related to the detection object. Specifically, the server first obtains the detection image collected in the image acquisition device, and then the server receives the detection event sent by the mobile terminal. The detection event is set by the user according to his or her specific needs, such as detecting whether a pet enters a specific area. This personalized setting method enables the monitoring method to meet the user's monitoring needs for different complex behaviors or different objects in the home scene. Using the multimodal large model to determine whether the detection image has the detection object in the detection event and the object behavior related to the detection object, the corresponding intelligent detection can be achieved based on the user's personalized or customized settings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 An application environment diagram of a home scene monitoring method supporting customized detection events in one embodiment;

[0039] Figure 2 A schematic diagram of a flow chart of a home scene monitoring method supporting customized detection events in one embodiment;

[0040] Figure 3 A schematic diagram of a process for determining whether a pet at home has a corresponding object behavior in one embodiment;

[0041] Figure 4 A schematic diagram of a flow chart of determining whether a child at home has a corresponding object behavior in one embodiment;

[0042] Figure 5 is a schematic diagram of determining whether a child is on a windowsill in one embodiment;

[0043] Figure 6 A schematic diagram of a process for generating a detection report using a multimodal large model in an embodiment;

[0044] Figure 7 A schematic diagram of a process of recording each detection combination and generating a detection report for each detection combination using a multimodal large model in one embodiment;

[0045] Figure 8 A schematic diagram of a specific flow chart of a home scene monitoring method supporting customized detection events in one embodiment;

[0046] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] like Figure 1 The monitoring system shown is used for monitoring home scenes. The home scene monitoring system is used to understand the situation of a specific area locally and remotely, and can monitor the real-time situation of key areas around the residence. The monitoring system includes a terminal 101, a server 102, and an image acquisition device 103. The home monitoring method supporting customized detection events provided in the embodiment of the present application is applied to the server 102. The terminal 101 can communicate with the server 102 via a network. Figure 1The data storage system in can be integrated on the server 102, or it can be placed on the cloud or other network servers. The user sets the detection event through the terminal 101, and the terminal 101 sends the detection event set by the user to the server 102. The server 102 obtains the detection image collected by the image acquisition device 103, determines whether there is a detection object and the object behavior related to the detection object based on the multimodal large model in the server, and then sends the detection result to the terminal 101. Among them, the terminal 101 can be but not limited to various personal computers, laptops, smart phones, tablet computers, wearable smart devices, etc. The server 102 can be implemented with a single server or a server cluster composed of multiple servers. The image acquisition device can be but not limited to a fixed camera, a network camera, a wireless camera, a rotatable camera, a panoramic camera, etc.

[0049] In one embodiment, Figure 2 As shown, a home scene monitoring method supporting custom detection events is provided, which is applied to the server side of the monitoring system, and the server side is configured with a multi-modal large model; the monitoring system also includes a mobile terminal and an image acquisition device that communicate with the server side, and the detection event includes at least one detection object and at least one object behavior related to the detection object. The following steps are included:

[0050] Step 201: Acquire a detection image acquired by an image acquisition device;

[0051] Specifically, the server obtains the detection image collected by the image acquisition device, and the collected detection image can be a real-time image captured by a camera. It should be noted that in this embodiment, the requirements for the camera are not high, and any camera that can provide stable shooting can meet the needs of the monitoring system.

[0052] Step 202: receiving a detection event sent by a mobile terminal, and using a multimodal large model to determine whether the detection image contains a detection object; and using the detection image containing the detection object as a target image;

[0053] Specifically, the server receives a detection event sent by a mobile terminal, and the detection event may refer to a detection object and at least one object behavior related to the detection object. For example, the puppy at home goes on the sofa, wherein the puppy at home is the detection object, and going on the sofa is an object behavior related to it; the kitten drinks water and eats cat food, wherein the kitten is the detection object, and drinking water and eating cat food are both object behaviors corresponding to it. It should be noted that the detection object in the detection event can be transmitted to the server by the mobile terminal through text description, image or other means. If the detection object in the detection event is represented in the form of an image, after the server obtains the detection image captured by the image acquisition device, the image of the detection object in the detection event and each frame of the detection image captured by the image acquisition device are calculated using a multimodal large model to determine whether the detection image captured by the image acquisition device contains the image of the detection object in the detection event, and the multiple frames of images containing the detection object in the detection image are used as target images. For example, the detection object in the detection event is a pet at home. If a frame of detection image captured by the acquisition device contains an image of the pet at home, then the detection image captured by the acquisition device can be used as the target image; if a frame of detection image captured does not contain an image of the pet at home, then the detection image captured at this time cannot be used as the target image.

[0054] It should be noted that the multimodal large model is an artificial intelligence technology that can process and understand many different types of data inputs, such as text, images, audio, and video. This model learns how to identify and match different types of data through deep learning and large-scale data training. When identifying whether there is a specific detection object in the target image, if the detection event contains a text description, the multimodal large model will use the text encoder to process these descriptions and understand their semantic content; if the image data of the detection object is directly included, the model will extract image features through the visual encoder. Then, the model uses modal fusion technology, such as the visual text alignment layer, to combine image features with text features to achieve cross-modal reasoning. In this way, the model can extract features from a large number of images collected by the image acquisition device after the user inputs the detection event, and perform reasoning and comparison with the detection event to accurately determine whether the detection image contains the detection object.

[0055] Step 203: Using the multimodal large model, determine whether the target image has the corresponding object behavior.

[0056] Specifically, the target image is analyzed using a multimodal large model to determine whether there is an object behavior related to the detection object in the target image. For example, if the object behavior related to the detection object is a dog at home getting on the sofa, if the dog does not have the behavior of getting on the sofa in the target image, then the detection object has not performed the object behavior at this time; only when the target image and the object behavior exist at the same time, such as the dog's behavior of getting on the sofa in the target image, can it be determined that the detection object and the object behavior related to the detection object exist.

[0057] In a home scene monitoring method supporting a custom detection event provided in the above embodiment, the server first obtains the detection image collected in the image acquisition device, and then the server receives the detection event sent by the mobile terminal. Among them, for the image acquisition device, only an ordinary camera is needed to complete the shooting needs required by the system, so the cost requirement for the image acquisition device is not very high. The detection event is set by the user according to his or her specific needs, such as detecting whether a pet enters a specific area. This personalized setting method makes the monitoring method more in line with the actual needs of the user. Further, a multimodal large model is used to determine whether the detection image contains a detection object, that is, the image acquired by the image acquisition device contains the detection object in the detection event, and the detection image containing the detection object is used as the target image. The multimodal large model is then used to determine whether the detection image contains the detection object in the detection event and the object behavior related to the detection object. Further, in this embodiment, this monitoring method provides users with a more reliable monitoring experience through user personalized settings and intelligent detection, without excessive requirements for image acquisition devices and without additional hardware costs.

[0058] In one embodiment, the detection object is an object whose position coordinates can change in a home scene; acquiring a detection image of an image acquisition device includes the following steps:

[0059] In response to an event instruction of an image acquisition device detecting an object movement event, a capture instruction is sent to the image acquisition device, so that the image acquisition device saves the detection image in response to the capture instruction and uploads the detection image to the server; wherein the image acquisition device turns on the motion detection function in response to the start instruction sent by the mobile terminal, and sends an event instruction to the server when an object movement event is detected.

[0060] Specifically, when the image acquisition device detects that an object is moving, the image acquisition device will send an event instruction to the server. When the server receives the instruction, it will send a capture instruction to the image acquisition device. At this time, the image acquisition device will start to capture images after receiving the instruction, save the captured images and upload them to the server. It should be noted that the user can send an instruction to turn on the motion detection function to the image acquisition device through the mobile terminal in advance. At this time, the image acquisition device will send an event instruction to the server as long as it detects the movement of an object.

[0061] In this embodiment, when the image acquisition device detects the movement of an object, it will send an event instruction to the server. When the server receives the instruction, it will send a capture instruction to the image acquisition device. After receiving the instruction, the image acquisition device will start to capture the image of the current scene and save it, and then upload the saved image to the server. This process will capture and save the image when an actual moving object is detected, and upload the saved image to the server, thereby saving the storage space of the image acquisition device and the resource space of the server.

[0062] In one embodiment, Figure 3 As shown, the detection object is a pet in a home scene, and the object behavior is that the pet moves to a specific object in the home scene. Using a multimodal large model, it is determined whether there is a corresponding object behavior in the target image, including the following steps:

[0063] Step 301: determining a recognition frame of a pet and a recognition frame of a specific object in a target image based on a multimodal model, wherein the multimodal model includes a pet recognition model;

[0064] Specifically, the server uses the pet recognition model in the multimodal large model, where the pet recognition model can recognize multiple pets and generate recognition frames for them. The server can also use the multimodal large model to obtain the recognition frame of a specific object. The recognition frame is usually determined by the coordinates of a rectangle, usually including the coordinates of the upper left corner and the lower right corner of the rectangle. For example, the object behavior of the target image is a kitten running to the dining table, where the kitten is a pet and the dining table is a specific object. The server can use the specific kitten recognition model of the pet recognition model in the multimodal large model to identify the kitten and generate a recognition frame for it, and use the multimodal large model to generate a recognition frame for the dining table.

[0065] Step 302: obtaining coordinates of two recognition frames, and determining an intersection area between the two recognition frames based on the coordinates of the two recognition frames;

[0066] Specifically, the server uses the multimodal large model to calculate and obtain the identification frames of the pet and the specific object, as well as the upper left corner and lower right corner coordinates of the two identification frames. Based on the coordinates corresponding to the two identification frames, the intersection of the two identification frames can be obtained. At this time, the intersection of the two identification frames is the intersection area between the two identification frames.

[0067] Step 303: If the area of ​​the intersection region is greater than the set threshold, then object behavior exists in the target image.

[0068] Among them, according to the example of the above steps, the server uses a multimodal large model to calculate the area of ​​the intersection area of ​​the kitten identification frame and the dining table identification frame based on the coordinates of the intersection area of ​​the two identification frames. After the intersection area result is calculated, it is compared with the threshold set by the user. If the intersection area of ​​the kitten and the dining table's identification frames is greater than the set threshold, it means that there is an intersection relationship between the kitten and the dining table, that is, there is an object behavior between the kitten and the dining table; if the intersection area of ​​the kitten and the dining table's identification frames is less than the set threshold, it cannot be said that there is a position intersection relationship between the kitten and the dining table, that is, it cannot be said that there is an object behavior between the kitten and the dining table.

[0069] In this embodiment, the server can accurately identify the identification frames of the pet and the specific object by using the multimodal large model, thereby improving the accuracy of detection. At the same time, the server can obtain the coordinates of the two identification frames using the multimodal large model and calculate the area of ​​the intersection of the two identification frames. By user-defined thresholds for the area of ​​the intersection area, the server can efficiently determine whether the pet intersects with the specific object, reducing the possibility of missed reports and false reports.

[0070] In one embodiment, Figure 4 As shown, the detection object is a child in a home scene, the object behavior is that the child moves to a specific object, and the specific object is an object with fixed position coordinates in the home scene; using a multimodal large model, determining whether there is a corresponding object behavior in the target image includes the following steps:

[0071] Step 401: determining a recognition frame of a child and a recognition frame of a specific object in a target image based on a multimodal large model, wherein the recognition frame of the child is used as a first recognition frame and the recognition frame of the specific object is used as a second recognition frame;

[0072] Specifically, in a home scene, parents need to monitor the interaction between their children and certain specific objects to ensure their safety. For example, a child standing on a windowsill is an object behavior related to the target image. The server uses the child recognition model in the multimodal large model to generate a recognition frame for the child in the target image, namely the first recognition frame. Similarly, the server uses the specific object recognition model in the multimodal large model to generate a recognition frame for a specific object (here the windowsill), namely the second recognition frame. The obtained recognition frames of the child and the specific object usually include the coordinates of the upper left corner and the lower right corner of the rectangle.

[0073] Step 402: if the first identification frame and the second identification frame have an intersection, determine whether the center of the first identification frame is located above the center of the second identification frame;

[0074] Specifically, the server uses a multimodal large model to calculate and obtain a first identification frame of the child and a second identification frame of a specific object, where the specific object may refer to a windowsill. The intersection range between the two identification frames is calculated based on the coordinates of the two identification frames, and it can be determined whether there is an intersection between the two positions. Only when there is an intersection between the two identification frames can it be determined that the child and the windowsill have a positional intersection relationship. In this example, the child standing on the windowsill is a behavior related to the target image and its object. The behavior emphasized here is that the child is on the windowsill, so it is necessary to determine whether the center of the child is located at the center of the windowsill, that is, whether the center of the first identification frame is at the center of the second identification frame. Among them, the calculation of the center coordinates of the first identification frame and the second identification frame can be done by the formula Here x min ,y max is the coordinate of the upper left corner of the recognition box, x max ,y min is the coordinate of the lower right corner of the recognition box, (x c ,y c ) is the center point of the identification frame. Only when the center point coordinates of the first identification frame are above the center point coordinates of the second identification frame can it be said that there is an object behavior of a child on the windowsill.

[0075] Step 403: If yes, determine whether the aspect ratio of the first recognition frame is less than a preset ratio;

[0076] Specifically, according to the above steps, the purpose of determining the length-to-width ratio of the first identification frame is to determine the state of the child's movements. When a child is on the windowsill, standing is generally more likely to cause danger than lying. Therefore, the server needs to use a multimodal large model to calculate the coordinates of the first identification frame to obtain the length-to-width ratio of the first identification frame, where the length-to-width ratio of the first identification frame can be used To calculate, where x min ,y max is the coordinate of the upper left corner of the first recognition box, x max ,ymin is the coordinate of the lower right corner of the first recognition frame. The preset ratio of length and width here can be 1, x max -x min Represents the length, y max -y min Represents width. When the length-to-width ratio of the first identification frame is less than the preset ratio, the width of the first identification frame of the child is greater than the length, that is, the child may be standing and is likely to engage in dangerous behavior on the windowsill; if the length-to-width ratio of the first identification frame is greater than the preset ratio, the width of the first identification frame of the child is less than the length, that is, the child may be lying down and is unlikely to engage in dangerous behavior on the windowsill.

[0077] Step 404: If yes, determine that the target image has object behavior.

[0078] Specifically, through the above steps, it is determined that the target image contains the object behavior related to the target image. That is, in the above example, if the center of the first identification frame representing the child is above the center of the second identification frame representing the windowsill, and the length-to-width ratio of the first identification frame of the child is less than the preset ratio, it means that the target image contains the object behavior of the child standing on the windowsill, and the server will send a notification to the mobile terminal.

[0079] It should be noted that if Figure 5 For example, in the detection event of a child standing on a windowsill, if the child and the windowsill have a position intersection relationship, and the center of the child is above the center of the windowsill, it can be determined that the child is standing on the windowsill. When the two recognition frames intersect and the intersection area is greater than the set threshold, it means that the two have a position intersection relationship. Secondly, the up-down relationship between the two center points can be calculated based on the coordinates of the two recognition frames, which can be the formula To calculate. Where x min ,y max is the coordinate of the upper left corner of the recognition box, x max ,y min is the coordinate of the lower right corner of the recognition box, (x c ,y c ) is the center point of the identification frame. When the identification frames of the child and the windowsill intersect and the center point coordinates of the child are above the center point coordinates of the windowsill, the preset ratio of length and width can be 1, x max -x min Represents the length, y max -y min If the length-to-width ratio of the first recognition frame is less than the preset ratio, it means that there is a child standing on the windowsill, and the server will send a notification to the mobile terminal.

[0080] In this embodiment, the server uses a multimodal large model to determine the identification frame of the child and the windowsill and determines whether the center of the child identification frame is located above the center of the windowsill identification frame, and determines whether the length-to-width ratio is less than a preset ratio to determine whether the child is standing on the windowsill. When each condition is met, it is determined that there is an object behavior, which effectively reduces the possibility of misjudgment. If the child does have an object behavior with a specific object, the server can immediately send a notification to the mobile terminal, so that the user can promptly discover the potential safety hazards that the child may face.

[0081] In one embodiment, obtaining a detection event sent by a mobile terminal includes: in response to a detection request sent by the mobile terminal, parsing the detection request to determine a detection event sent by the mobile terminal; wherein the detection event includes not only a detection object and an object behavior, but also a cycle for generating a detection report; and the method further includes:

[0082] If the detection image contains the detection object, the server sends a notification to the mobile terminal, and when the cycle for generating the detection report is reached, the server uses a multimodal large model to analyze the detection results of the object behavior in the target image within the period and generate a detection report, which is sent to the mobile terminal.

[0083] Specifically, a user can send a detection event through a mobile terminal, which includes a detection object, an object behavior related to the detection object, and a cycle for generating a detection report. The server receives the detection event from the mobile terminal and parses it to determine the specific content and requirements of the detection event, i.e., the detection object, the object behavior, and the cycle for generating a detection report. The server uses a multimodal large model to determine whether each frame of the detection image obtained by the image acquisition device contains the detection object. If the detection object is contained, the corresponding frame detection image containing the detection object is used as the target image and a notification is sent to the mobile terminal. The server sets a monitoring cycle according to the analysis of the detection event content and starts monitoring at the beginning of each cycle. In each cycle, the server continuously obtains the detection image transmitted by the image acquisition device, and the server uses the multimodal large model to determine which frames in the detection image collected by the image acquisition device contain the detection object and the object behavior related to it, and records the time when the object behavior occurs. At the end of each monitoring cycle, the server uses the multimodal large model to analyze the detection results in the cycle and generate a detailed detection report. The server sends the detection report to the user's mobile terminal, and the user can understand the detection object and the occurrence of the related object behavior in the user-defined detection event within the cycle time.

[0084] In this embodiment, the user can receive notifications sent by the server in real time through the mobile terminal, and the user can receive periodic detailed reports of custom detection events on a regular basis. This helps users better understand and monitor the safety status of their homes. The server can automatically collect and analyze data, reducing the user's analysis of a large number of detection results and improving monitoring efficiency. At the same time, the user can understand the behavioral habits of the detection object, such as children or pets, by analyzing the detection results within the cycle, providing effective data support for family education and safety prevention.

[0085] In one embodiment, Figure 6 As shown, using a multimodal large model, the detection results of the object behavior in the target image within the period are analyzed and a report is generated, including the following steps:

[0086] Step 601: Based on the detection object and the object behavior, use the multimodal large model to identify whether the detection object in the target image has the object behavior, and if so, record the occurrence time of the object behavior;

[0087] Specifically, the user sets monitoring parameters through the mobile terminal, including detection objects, object behaviors, and monitoring periods. For example, from 8 to 10 a.m. every day, whether the children at home are close to the power strip, where the detection object is the child, the object behavior is close to the power strip, and the monitoring period is from 8 to 10 a.m. every day. During the set monitoring period, the image acquisition device continuously captures images in the monitoring area. After the server receives the image uploaded by the image acquisition device, it uses the multimodal large model to analyze and identify the target image, and determines whether the detection object in the target image has a corresponding object behavior, that is, whether the image of the child in the example has the behavior of approaching the power strip. If the multimodal large model determines that there is a corresponding object behavior, the server will record the time when the object behavior occurs. That is, if the child approaches the power strip at 9:25 in the morning, the time will be recorded by the server.

[0088] Step 602: until the period for generating the detection report is met, the multimodal large model analyzes and generates a detection report based on the recorded occurrence time, detection object and object behavior.

[0089] Specifically, until the period for generating a detection report is met, that is, from the beginning to the end of the monitoring period, the multimodal large model analyzes the collected data, including the time and frequency of the object's behavior. The server can generate a detection report based on the analysis results, which contains the most detailed statistical data on the object's behavior, such as the time and number of times the child approaches the power strip, and the time period when the child most frequently approaches the power strip.

[0090] In the present embodiment, the monitoring system can accurately record the time when the behavior of the detection object and its corresponding object occurs, and provide detailed data for the subsequent generation of periodic reports. Through the generated periodic detection reports, users can regularly understand the behavior patterns of the detection objects, so that users do not need to monitor continuously for a long time, which improves the efficiency of monitoring. The detection report generated by the multimodal large model can contain key information such as the frequency time period distribution of the behavior, helping users to better understand the behavioral habits of the detection object. Users can customize the monitoring cycle and behavior according to their needs, making the system more convenient and meeting the needs of different groups of people.

[0091] In one embodiment, Figure 7 As shown, in a unit time, the server segment needs to detect multiple detection objects and multiple object behaviors sent by the mobile terminal, and one detection object manages at least one object behavior; using a multimodal large model, the detection results of the object behavior in the target image within the period are analyzed and a detection report is generated, including the following steps:

[0092] Step 701: taking an object behavior expected to be associated with the detection object as a detection combination, for each detection combination, using the multimodal large model to identify whether the detection object of the detection combination in the detection image has the object behavior of the detection combination, and if so, recording the time when the object behavior occurs;

[0093] Specifically, within a unit of time, the server needs to detect multiple detection objects and multiple object behaviors sent by the mobile terminal, and one detection object manages at least one object behavior, and each detection object and its related object behavior form a detection combination. The user can send, for example, a kitten eating cat food, a kitten on the dining table, a child climbing on the sofa, an old man going home, etc. Among them, the kitten can correspond to two object behaviors of drinking water and going to the dining table, the child corresponds to an object behavior of going on the sofa, and the old man corresponds to an object behavior of going home. These objects and object behaviors are combined into a detection combination, where the kitten drinking water and the kitten going to the dining table are two detection combinations. The server uses a multimodal large model to determine whether there is an object behavior related to the detection object in the detection combination in the target image. If so, the time when the object behavior in the detection combination occurs is recorded.

[0094] Step 702: until the cycle for generating the detection report is met, for each detection object, determine the occurrence time of the object behavior associated with the detection object, and the multimodal large model generates a detection report based on the detection object and the occurrence time of the associated object behavior.

[0095] Specifically, until the detection cycle time ends, the server uses the multimodal large model for each detection object in each detection combination and its related object behavior, and all records the time when its object behavior occurs. Secondly, the server uses the multimodal large model to generate a detection report based on the detection object and the associated object behavior occurrence time. That is, according to the example in step 701, the multimodal large model is used to generate a detection report for the object behavior occurrence time recorded in the detection combination such as kitten eating cat food, kitten on the dining table, child climbing on the sofa, and the elderly going home. It should be noted that the monitoring period of each detection combination can be different. For example, the detection objects such as kitten eating cat food, kitten on the dining table, and child climbing on the sofa can be the monitoring period from 9 am to 6 pm during working hours, and the elderly going home can be the elderly walking time from 9 am to 11 am as the monitoring period. For each different detection combination, the server can record the corresponding behavior occurrence time and generate a detection cycle report corresponding to each detection combination.

[0096] In this embodiment, by forming a detection combination of different detection objects and at least one object behavior corresponding to them, the server can process multiple detection objects and behaviors at the same time, so the server can use computing resources more efficiently, reduce repeated detection and analysis processes, and monitor multiple behaviors of each detection object in a comprehensive and detailed manner. For each detection combination, the server can accurately record the occurrence time of the object behavior, providing accurate data support for subsequent analysis, and the generated detection report can be used as data support to help users make more reasonable response decisions based on actual object behavior.

[0097] In one embodiment, a monitoring method for a home scene supporting a custom detection event is applied to a mobile terminal of a monitoring system, wherein the monitoring system further includes a server and an image acquisition device, and the method includes:

[0098] Determine a user-defined detection event and send the detection event to the server so that the server can execute the above-mentioned embodiments based on the detection event and the detection image captured by the image acquisition device, wherein the detection event includes at least one detection object and at least one object behavior related to the detection object.

[0099] Specifically, the user sends a user-defined detection event to the server through a mobile terminal, and the detection event includes at least one detection object and at least one object behavior related to the detection object and a monitoring period. After receiving the detection event, the server analyzes it and stores it in a database for subsequent processing. After receiving the image uploaded by the image acquisition device, the server uses a multimodal large model to analyze and identify the target image, and determines whether there is an object behavior of the detection object in the target image. If a user-defined detection event is detected, the server will record it and send a notification to the mobile terminal. The server can also generate a periodic report based on the recorded detection event data after the set monitoring period ends. The report is sent to the user through the mobile terminal, which contains the frequency of object behavior within the period and the time period when the behavior occurs.

[0100] In this embodiment, through the home scene monitoring method, users can customize detection events according to their needs and family situations, making the monitoring system more personalized, and can also receive notifications of home detection events in a timely manner to improve their understanding of home situations. Through periodic reports, users can understand the behavior patterns of family members and pets, providing data support for family education and pet management. At the same time, the high accuracy of the multimodal large model is used to reduce the possibility of false alarm notifications.

[0101] In one embodiment, Figure 8 As shown, the home scene monitoring method supporting custom detection events includes the following specific processes:

[0102] Step 801: The mobile terminal sends a motion detection start instruction to the image acquisition device;

[0103] Step 802: The mobile terminal sends the custom detected object and object behavior to the server;

[0104] Step 803: The mobile terminal sends a customized cycle for generating a detection report to the server;

[0105] Step 804: the server responds to the event instruction of the image acquisition device detecting the object movement event;

[0106] Step 805: the server sends a capture instruction to the image acquisition device;

[0107] Step 806: the image acquisition device saves the detection image and uploads the detection image to the server;

[0108] Step 807: The server analyzes the detection object set by the user and the image uploaded by the camera based on the multimodal large model;

[0109] Step 808: The server sends the detection result including the detection object set by the user to the mobile terminal;

[0110] Step 809: The user views the daily test results sent by the server through the mobile terminal;

[0111] Step 810: When the reporting period set by the user is reached, the detection results of the recent period are classified and sorted based on the multimodal large model, and suggestions are given;

[0112] Step 811: The user views the periodic detection report sent by the server through the mobile terminal.

[0113] In this embodiment, an efficient and customizable monitoring solution is realized by integrating image acquisition devices, mobile terminals and cloud servers. Users can set detection events according to their needs, and the system automatically performs monitoring and analysis tasks, notifies users of important events in a timely manner, and provides periodic reports, thereby enhancing home safety and convenience.

[0114] Based on the same inventive concept, an embodiment of the present application further provides a monitoring system for implementing the above-mentioned home scene monitoring method supporting customized detection events. Specifically, a monitoring system supporting customized detection events includes:

[0115] One or more image acquisition devices configured to acquire detection images in a home scene;

[0116] One or more mobile terminals, configured to obtain user-defined detection events;

[0117] The cloud server is configured to communicate with the image acquisition device and the mobile terminal to execute the various steps in the embodiment of the above-mentioned home scene monitoring method supporting customized detection events.

[0118] Specifically, one or more cameras are installed in the home environment as image acquisition devices, and these devices are configured to collect images in the home scene in real time. The user defines detection events on the mobile terminal, such as monitoring whether the elderly at home fall or whether the pet enters a specific area, and sends these events to the cloud server. The user turns on the motion detection function of the camera to the image device collector through the mobile terminal, and sends an event instruction to the server after detecting the movement event of the object. When the server receives the instruction, it sends a capture instruction to the image acquisition device, and the image acquisition device continues to collect images and continuously transmits the images to the cloud server. The cloud server receives the image data, analyzes the image using a multimodal large model, and identifies the detection objects and behaviors defined by the user. When the server detects a behavior that matches the user-defined detection event, it sends a record and notifies the user's mobile terminal. The cloud server can also regularly generate a report containing the analysis of the detection event, and provide it to the user through the mobile terminal so that the user can understand the long-term trend of the monitoring event.

[0119] In one embodiment, an electronic device is further provided. The electronic device is applied to a server side. The electronic device may also be a server. Specifically, the electronic device includes:

[0120] Memory for storing computer programs;

[0121] The processor is used to execute the computer program. When the computer program is continuously executed by the processor, each step in the above embodiment is implemented.

[0122] The internal structure diagram of the electronic device can be Fig. 9 As shown, the electronic device includes a memory and a processor, the memory is used to store computer programs, and the processor is used to execute these programs. In the memory, a series of computer programs are stored, which contain codes for implementing the monitoring system, such as image acquisition, user-defined event processing, multi-modal large model analysis, etc. The processor loads and executes these computer programs from the memory to implement each step. When the processor executes the image acquisition program, it controls the image acquisition device to collect images in the home scene. When executing the user-defined event processing program, the processor receives the user-defined detection event from the mobile terminal and configures the monitoring parameters according to these events. When executing the multi-modal large model analysis program, the processor analyzes the collected image data and identifies the detection object and behavior. In addition, the processor also generates a detection report according to the monitoring cycle set by the user, and sends the report to the terminal to feedback to the user so that the user can understand the general trend within the monitoring cycle.

[0123] In this embodiment, the electronic device realizes an efficient and automated monitoring system through the cooperative work of its memory and processor. It can process a large amount of image data, identify and respond to user-defined detection events, and provide timely notifications and periodic reports, thereby improving the intelligence level of home monitoring and enhancing the user's monitoring experience. At the same time, the device is also flexible and can perform different monitoring tasks according to the specific needs of the user.

[0124] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0125] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A home scene monitoring method supporting custom detection events, characterized in that: The method is applied to a server side of a monitoring system, wherein the server side is configured with a multimodal large model; the monitoring system further comprises a mobile terminal and an image acquisition device communicating with the server side, wherein the mobile terminal is configured to obtain a user-defined detection event, wherein the detection event comprises at least one detection object and at least one object behavior related to the detection object; and the method comprises: Acquire the detection image acquired by the image acquisition device; receiving a detection event sent by the mobile terminal, and determining whether the detection image contains the detection object by using the multimodal large model; and using the detection image containing the detection object as a target image; The multimodal large model is used to determine whether there is a corresponding object behavior in the target image, and if so, the mobile terminal is notified.

2. The home scene monitoring method according to claim 1, characterized in that: The detection object is an object whose position coordinates in the home scene can change; Acquiring the detection image acquired by the image acquisition device includes: In response to an event instruction that the image acquisition device detects an object movement event, a capture instruction is sent to the image acquisition device, so that the image acquisition device saves the detection image in response to the capture instruction and uploads the detection image to the server; wherein the image acquisition device turns on the motion detection function in response to the start instruction sent by the mobile terminal, and sends an event instruction to the server when an object movement event is detected.

3. The home scene monitoring method according to claim 1, characterized in that: The detection object is a pet in the home scene, and the object behavior is the pet moving to a specific object in the home scene; Using the multimodal large model, determining whether a corresponding object behavior exists in the target image includes: Determining a recognition frame of the pet and a recognition frame of the specific object in the target image based on the multimodal model, wherein the multimodal model includes a pet recognition model; Obtaining coordinates of the two recognition frames, and determining an intersection area between the two recognition frames based on the coordinates of the two recognition frames; If the area of ​​the intersection region is greater than a set threshold, then object behavior exists in the target image.

4. The home scene monitoring method according to any one of claim 1, characterized in that: The detection object is a child in a home scene, the object behavior is the child moving to a specific object, and the specific object is an object with fixed position coordinates in the home scene; Using the multimodal large model, determining whether a corresponding object behavior exists in the target image includes: Determining a recognition frame of the child and a recognition frame of the specific object in the target image based on the multimodal large model, wherein the child recognition frame is used as a first recognition frame and the specific object recognition frame is defined as a second recognition frame; If the first identification frame and the second identification frame have an intersection area, determining whether the center of the first identification frame is located above the center of the second identification frame; If so, determining whether the aspect ratio of the first identification frame is less than a preset ratio; If so, it is determined that object behavior exists in the target image.

5. The home scene monitoring method according to any one of claims 1 to 4, characterized in that: Acquiring the detection event sent by the mobile terminal includes: in response to the detection request sent by the mobile terminal, parsing the detection request to determine the detection event sent by the mobile terminal; wherein the detection event includes not only the detection object and the object behavior, but also the cycle of generating the detection report; The method further comprises: If the detection image contains a detection object, the server sends a notification to the mobile terminal, and when the cycle for generating a detection report is reached, the server uses the multimodal large model to analyze the detection results of the object behavior in the target image within the cycle and generate a detection report, which is sent to the mobile terminal.

6. The home scene monitoring method according to claim 5, characterized in that: Utilizing the multimodal large model, analyzing the detection results of the object behavior in the target image within the period and generating a detection report, including: Based on the detection object and the object behavior, using the multimodal large model to identify whether the detection object in the target image has the object behavior, and if so, recording the occurrence time of the object behavior; Until the period for generating the detection report is met, the multimodal large model analyzes and generates a detection report based on the recorded occurrence time and the detection object and the object behavior.

7. The home scene monitoring method according to claim 6, characterized in that: The server needs to detect multiple detection objects and multiple object behaviors sent by the mobile terminal within a unit time, and one detection object is associated with at least one object behavior; Utilizing the multimodal large model, analyzing the detection results of the object behavior in the target image within the period and generating a detection report, including: The detection object and an object behavior associated with it are taken as a detection combination. For each detection combination, the multimodal large model is used to identify whether the detection object of the detection combination in the detection image has the object behavior of the detection combination. If so, the occurrence time of the object behavior is recorded; Until the cycle for generating the detection report is met, for each detection object, the occurrence time of the object behavior associated with the detection object is determined, and the multimodal large model generates a detection report based on the occurrence time of the detection object and the associated object behavior.

8. A home scene monitoring method supporting custom detection events, characterized in that: A mobile terminal applied to a monitoring system, wherein the monitoring system further comprises a server and an image acquisition device, wherein the image acquisition device is used to acquire detection images in a home scene, and the method comprises: Determine a user-defined detection event and send the detection event to a server side, so that the server side can execute the method as described in any one of claims 1 to 7 based on the detection event and the detection image acquired by the image acquisition device, wherein the detection event includes at least one detection object and at least one object behavior related to the detection object.

9. A monitoring system supporting custom detection events, characterized in that: The monitoring system comprises: One or more image acquisition devices configured to acquire detection images in a home scene; One or more mobile terminals, configured to obtain user-defined detection events; A cloud server is configured to communicate with the image acquisition device and the mobile terminal to execute the steps of the method according to any one of claims 1 to 7.

10. An electronic device, applied to a server, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images