Surveillance system capable of automatically generating text summaries and method thereof

Abstract

A monitoring system and method capable of automatically generating a text summary are provided. The present invention provides a surveillance system capable of automatically generating text summaries, including a portable electronic device, an image capture device, a computing device, and a signal transmission device. The image capture device is used to capture images of a monitored area. The computing device is capable of reading and executing a stored intelligent application image recognition program, processes the images of the monitored area, generates corresponding text summaries, and transmits the text summaries to the portable electronic device via the signal transmission device.

Description

[Technical Field]

[0001] The present invention relates to an environmental monitoring system, and more particularly to a smart monitoring system that generates specific information. [Background technology]

[0002] With the declining birthrate and the aging society, families who are away from home for work may need to use a home security monitoring system to check on the status of children and elderly people at home. Home security monitoring systems not only effectively deter potential intruders, but also provide real-time monitoring and evidence collection when safety incidents occur. Summary of the Invention [Problem to be solved by the invention]

[0003] Typically, family members who are away from home for work use images captured by a home security monitoring system to monitor the status of their home or surroundings. However, frequently reviewing large amounts of surveillance footage is time-consuming and mentally taxing, impacting work efficiency, and transmitting large amounts of surveillance footage can lead to a dramatic increase in network traffic. In prior art, an electronic fence is set up within the surveillance area, and an abnormality alert is generated when someone enters the electronic fence. Users then view the video of the alert to confirm the abnormality and determine whether the abnormality poses a risk. However, while the electronic fence in prior art effectively reduces the time and frequency with which users view surveillance footage, statistics show that 50% of abnormality alerts generated by electronic fences are harmless.

[0004] Therefore, the technical problem to be solved by the present invention is how to provide a safety monitoring system that allows a user to quickly understand the status of the monitored area.

[0005] The present invention has been made in view of the above circumstances, and aims to solve the above problems, namely, a main object of the present invention is to provide a safety monitoring system that enables a user to quickly understand the status of a monitored area. [Means for solving the problem]

[0006] In order to achieve the above object, one aspect of the present invention is a monitoring system capable of automatically generating text summaries, comprising: A portable electronic device; an image capture device for capturing an image of the monitored area; A computing device storing an intelligent application image recognition program, the computing device comprising: (a) setting a range of a monitoring area and notification conditions; (b) operating an image capture device to capture an image of the monitored area; (c) determining whether the image being captured triggers a notification condition; (d) extracting video segments from the images that meet the notification criteria; (e) dividing the video segment into a series of image files; (f) extracting from each of the image files subjects and subject actions that meet the notification condition; (g) generating a passage of narrative text corresponding to each subject and each action of the plurality of image files and combining the passages of narrative text corresponding to the video segments in a chronological order to form a solitaire synopsis; a computing device that executes an intelligent application image recognition program; a signal transmission device that establishes a communication connection between the portable electronic device, the image capture device, and any one of the computing device; A signal transmission device transmits the solitaire synopsis from the computing device to the portable electronic device over the communications connection.

[0007] In a preferred embodiment of the present invention, in step (a), the notification conditions include an object identification feature and an object action recognition feature.

[0008] In a preferred embodiment of the present invention, in step (f), the computer device (f1) extracting entities that match the object identification features based on the object identification features; (f2) extracting an action that matches the object action recognition feature based on the object action recognition feature.

[0009] In a preferred embodiment of the present invention, in step (f), the computer device further executes the step of sequentially adding a timestamp to each of the image files.

[0010] In a preferred embodiment of the present invention, in step (g), the computer device (g1) generating a subject predicate sentence that matches the subject and generating an action predicate sentence that matches the action; (g2) forming a paragraph of predicate sentence by combining the subject predicate sentence and the action predicate sentence in chronological order.

[0011] In a preferred embodiment of the present invention, in step (e), the computer device (e1) dividing a video segment into a series of image files; (e2) adding a timestamp to each of the image files in turn.

[0012] In a preferred embodiment of the present invention, the computing device is a home personal computer or a cloud server.

[0013] In a preferred embodiment of the present invention, the intelligent application image recognition program includes an image recognition model database, which includes a plurality of object identification feature data and a plurality of object action recognition feature data, and which newly generates a plurality of object identification feature data and a plurality of object action recognition feature data based on training data input into a video segment.

[0014] To achieve the above object, another aspect of the present invention is a method capable of automatically generating a text summary from image capture, used to analyze images of a surveillance area captured by an image capture device, the method comprising: (a) setting a range of a monitoring area and notification conditions; (b) operating an image capture device to capture an image of the monitored area; (c) determining whether the image being captured triggers a notification condition; (d) extracting video segments from the images that meet the notification criteria; (e) dividing the video segment into a series of image files; (f) extracting from each of the image files subjects and subject actions that meet the notification condition; (g) generating a passage of narrative text corresponding to each subject and each action of the plurality of image files and combining the passages of narrative text corresponding to the video segments in a chronological order to form a solitaire synopsis; (h) transmitting the solitaire summary to the portable electronic device.

[0015] In a preferred embodiment of the present invention, in step (a), the notification conditions include an object identification feature and an object action recognition feature.

[0016] In a preferred embodiment of the present invention, in step (f), (f1) extracting entities that match the object identification features based on the object identification features; (f2) extracting an action that matches the object action recognition feature based on the object action recognition feature.

[0017] In a preferred embodiment of the present invention, step (f) further comprises the step of sequentially adding a timestamp to each of the image files.

[0018] In a preferred embodiment of the present invention, in step (g), the computer device (g1) generating a subject predicate sentence that matches the subject and generating an action predicate sentence that matches the action; (g2) forming a paragraph of predicate sentence by combining the subject predicate sentence and the action predicate sentence in chronological order.

[0019] In a preferred embodiment of the present invention, in step (e), the computer device (e1) dividing a video segment into a plurality of image files; (e2) adding a timestamp to each of the image files in turn. [Effects of the Invention]

[0020] The surveillance system capable of automatically generating text summaries of the present invention captures video segments when an alert condition is triggered, and generates a narrative passage and a solitaire summary corresponding to the video segment. A user can obtain the solitaire summary through a portable device and quickly understand the status of the monitored area from the information content of the solitaire summary, eliminating the need to spend a lot of time and effort monitoring the video.

[0021] At least the following points will become clear from the description and drawings to be described later. [Brief explanation of the drawings]

[0022] [Figure 1A] 1 illustrates a surveillance system capable of automatically generating text summaries according to a first embodiment of the present invention. [Figure 1B] 1 illustrates a surveillance system capable of automatically generating text summaries according to a second embodiment of the present invention. [Figure 2] 1 shows a process flowchart for automatically generating a text summary by image capture according to the present invention. [Figure 3] 3 is an exploded flowchart of step S100 in FIG. 2. [Figure 4] 3 is a flowchart illustrating an exploded view of step S104 in FIG. 2. [Figure 5] 3 is a flowchart illustrating an exploded view of step S105 in FIG. 2. [Figure 6] 3 is a flowchart illustrating an exploded view of step S106 in FIG. 2. [Figure 7A] 1 is a schematic diagram illustrating the operation of a surveillance system capable of automatically generating text summaries according to one embodiment of the present invention; [Figure 7B] 1 is an operational schematic diagram illustrating a monitoring system capable of automatically generating text summaries according to one embodiment of the present invention. [Figure 8] 1 is a system block diagram illustrating an intelligent application image recognition program according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0023] The following describes in detail the embodiments of the present invention, but the present invention is not limited to these, and various modifications are possible within the scope of the description, and embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention.

[0024] FIG. 1A illustrates a surveillance system 1 capable of automatically generating text summaries according to a first embodiment of the present invention. The surveillance system 1 includes a computing device 10, an image capture device 20, a signal transmission device 30, an intelligent application image recognition program 40, a cloud server 50, and a portable electronic device 60. The computing device 10 can establish a communication connection and an electrical connection between the image capture device 20 and the signal transmission device 30. The intelligent application image recognition program 40 is stored in a storage module or non-volatile memory (not shown) of the computing device 10. The computing device 10 reads and executes the intelligent application image recognition program 40 to perform a series of image processing and text generation steps on images captured or recorded by the image capture device 20.

[0025] In this embodiment, the computing device 10 is a home personal computer, the image capture device 20 is a video camera or a surveillance device, the signal transmission device 30 is a network bridge, a network switch, a router, a Bluetooth device, or a network switch, and the portable electronic device 60 is a smartphone or a tablet terminal.

[0026] The signal transmission device 30 establishes a communication connection between any one of the portable electronic device 60, the image capture device 20, and the computer device 10 via wired signal transmission or a communication protocol CP. The communication protocol CP may be a mobile network protocol (e.g., 4G or 5G), a wireless LAN transmission protocol (e.g., Wi-Fi), Bluetooth, or Internet Protocol (IP-based), etc. In some possible implementations, the signal transmission device 30 first executes an intelligent application image recognition program 40 on the computer device 10, and then uploads the image and text information generated in the processing to a cloud server 50 via the communication protocol CP, and transmits the image and text information generated by the cloud server 50 to the portable electronic device 60 in the form of a short message service (SMS), email, an app interface, etc. In this way, the user h reads the image and text information generated by the portable electronic device 60. In another possible implementation method, the signal transmission device 30 may transmit images and text information generated by the computing device 10 directly to the portable electronic device 60 via a communication protocol CP such as Bluetooth communication, without going through the cloud server 50.

[0027] 1B shows a second embodiment of a surveillance system capable of automatically generating text summaries according to the present invention. The surveillance system 1' includes an image capture device 20, a signal transmission device 30, an intelligent application image recognition program 40, a cloud server 50, and a portable electronic device 60. The intelligent application image recognition program 40 is stored in the cloud server 50. The signal transmission device 30 uploads images captured or recorded by the image capture device 20 to the cloud server 50 via a communication protocol CP. The cloud server 50 reads and executes the intelligent application image recognition program 40 and performs a series of image processing and text generation steps on the received images. The generated images and text information are then transmitted to the portable electronic device 60 via a short message service (SMS), email, or an app interface.

[0028] FIG. 2 shows a process flowchart for automatically generating a text summary by image capture according to the present invention. FIG. 3 is a decomposition flowchart of step S100 in FIG. 2. FIG. 4 is a decomposition flowchart of step S104 in FIG. 2. FIG. 5 is a decomposition flowchart of step S105 in FIG. 2. FIG. 6 is a decomposition flowchart of step S106 in FIG. 2. FIG. 7A is a schematic diagram illustrating the operation of a monitoring system capable of automatically generating a text summary according to an embodiment of the present invention. FIG. 7B is a schematic diagram illustrating the operation of a monitoring system capable of automatically generating a text summary according to an embodiment of the present invention.

[0029] First, the scope of the monitoring area Z and the notification conditions are set (step S100). In step S100, the image capture device 20 is installed in an environment to be monitored, such as the periphery of a residential environment or the product display area of ​​a store. The user h sets the scope of the monitoring area Z ((i) in FIG. 7A ) and the notification conditions for the image capture device 20 through an input interface, such as a keyboard or touch screen, of the computing device 10. In some possible implementations, the monitoring system 1 further includes special notification conditions that are set for certain special situations, such as a burglary, a fire, a dense smoke alarm, or an elderly person or a child falling. The special notification conditions are set in advance by the monitoring system 1 or manually by the user. When a special notification condition is triggered, the monitoring system 1 not only continuously performs subsequent image processing and text generation steps, but also automatically notifies the police, fire department, medical institution, security company, or user h of the monitoring system 1.

[0030] In the example of FIG. 3 , step S100 further includes step S1001 of setting the scope of the monitoring area Z, step S1002 of setting an object identification feature for the notification condition, and step S1003 of setting an object motion recognition feature for the notification condition. In step S1002, the object identification feature is used to determine whether an entity that meets the notification condition appears in the captured image. For example, the entity identified by the object identification feature related to a vehicle may be a passenger car, a recreational vehicle, a station wagon, a van, a coupe, or a convertible. In step S1003, the object motion recognition feature is used to determine whether an entity motion that meets the notification condition appears in the captured image, such as a vehicle entering, a vehicle stopping, a vehicle crashing, a vehicle overturning, a vehicle running out of control, a rollover, or a door opening or closing. In some possible implementations, the notification condition is not triggered unless both the object identification feature and the object motion recognition feature are met simultaneously, or the notification condition is triggered simply by meeting the object identification feature.

[0031] Next, the image capture device 20 is driven to capture an image of the monitoring area Z (step S101). In step S101, the driven image capture device 20 captures or records an image of the monitoring area Z throughout the day or during a predetermined time period.

[0032] Then, it is determined whether the captured image triggers a notification condition (step S102). In step S102, the computing device 10 reads and executes the intelligent application image recognition program 40 to analyze the image captured or recorded by the image capture device 20 to determine whether the image contains a subject and subject's behavior that meets the notification condition, such as an object identification feature or an object action recognition feature. If the determination result is "No," it continues to execute step S101. If the determination result is "Yes," it executes step S103, which extracts video segments that meet the notification condition from the image. In step S103, video segments that have the notification condition, such as an object identification feature or an object action recognition feature, are captured from the image of the monitoring area Z.

[0033] Next, the video segment is divided into a series of multiple image files (step S104). In step S104, image files in the video segment are cut and captured based on a predetermined time interval. In the example of FIG. 4, step S104 further includes step S1041 of dividing the video segment into multiple image files and step S1042 of sequentially adding a timestamp to each image file. In step S1041, if the predetermined time interval is 5 seconds, one image file is captured from the video segment every 5 seconds. In step S1042, if the predetermined time interval is 5 seconds, a timestamp is sequentially added to each captured image file, for example, 5S, 10S, 15S, and 20S. In subsequent processing, the timestamps of each image file are used as the basis for combining the subjects and actions of multiple image files in chronological order, for example, (1) a vehicle (subject) enters a yard and then stops (action), (2) the door of the vehicle (subject) opens (action), and (3) the driver of the vehicle (subject) gets out of the car (action).

[0034] Next, the subject and the subject's action that meets the notification condition are extracted from each image file (step S105). In step S105, the subject and the subject's action are extracted from the series of image files using the object identification feature and object action recognition feature of the notification condition.

[0035] 5, step S105 further includes step S1051 of sequentially adding a timestamp to each image file, step S1052 of extracting a subject that matches the object identification feature based on the object identification feature, and step S1053 of extracting a motion that matches the object motion recognition feature based on the object motion recognition feature. Step S1051 executes the same processing as step S1042, and is modified so that step S1051 of step S105 executes the processing of adding a timestamp if step S1042 of adding a timestamp has not been executed in the preceding step S104. Conversely, step S1051 of step S105 may be omitted if step S1042 of adding a timestamp has been executed in the preceding step S104.

[0036] Next, in step S1052, an entity corresponding to the object identification features is extracted. For example, when vehicle C enters monitoring area Z (see (ii) of FIG. 7A), an entity matching the object identification features of the vehicle is extracted based on the object identification features of the vehicle, i.e., vehicle C is extracted. After the entity to be extracted (vehicle C) is determined, in step S1053, an action matching the object action recognition features of the vehicle is extracted based on the object action recognition features of the vehicle, i.e., entry into monitoring area Z is extracted.

[0037] Next, a section of narrative sentence corresponding to each subject and each action of the plurality of image files is generated, and the sections of narrative sentence corresponding to the video segments are combined in chronological order to form a solitaire summary (step S106). In step S106, sections of narrative sentence of video segments corresponding to different time periods and having the same or different notification conditions are combined in chronological order to form a solitaire summary corresponding to a plurality of different time periods.

[0038] In the example of FIG. 6, step S106 further includes step S1061 of generating a subject predicate sentence that matches the subject and generating an action predicate sentence that matches the action, and step S1062 of combining the subject predicate sentence and the action predicate sentence in a time series to form a single predicate sentence. In step S1061, a subject predicate sentence that matches the subject (vehicle C) is generated, for example, "sedan." Next, an action predicate sentence that matches the action (entering monitoring area Z) is generated, for example, "entering the yard of the house." In step S1062, the subject predicate sentence (i.e., "sedan") and the action predicate sentence (i.e., "entering the yard of the house") are combined based on the time series to form a single predicate sentence (i.e., "the sedan enters the yard of the house") (see (ii) of FIG. 7A).

[0039] When the execution of step S106 is completed, return to step S102, and simultaneously or subsequently execute step S107 to transmit the solitaire summary to user h's portable electronic device 60. In this way, user h reads the summary information generated and transmitted by the computing device 10 or the cloud server 50 from the portable electronic device 60.

[0040] In the example of FIG. 7B, the computing device 10 (see FIG. 1A) or the cloud server 50 (see FIG. 1B) extracts, from the images acquired by the image capture device 20, a video segment VS1 (object identification features and object motion recognition features including a postman P1), a video segment VS2 (object identification features and object motion recognition features including an electric power company employee P2), a video segment VS3 (object identification features and object motion recognition features including a dog D), a video segment VS4 (object identification features and object motion recognition features including a female visitor P3, a vehicle C, etc.), and a video segment VS5 (object identification features and object motion recognition features including a young child P4) that meet the reporting conditions.

[0041] Next, using the processing flowchart shown in Figure 2, each of the video segments VS1, VS2, VS3, VS4, and VS5 is processed to generate a passage of narrative sentence corresponding to each of the video segments VS1, VS2, VS3, VS4, and VS5, and the passage of narrative sentences is combined in chronological order to form a solitaire summary, which is then transmitted to the portable electronic device 60 of the user h.

[0042] In this embodiment, the solitaire summary RA displayed on the portable electronic device 60 includes a passage of narrative text TS1 (corresponding to video segment VS1), a passage of narrative text TS2 (corresponding to video segment VS2), a passage of narrative text TS3 (corresponding to video segment VS3), a passage of narrative text TS4 (corresponding to video segment VS4), and a passage of narrative text TS5 (corresponding to video segment VS5). In this way, the user h can view the solitaire summary RA from the display interface of the portable electronic device 60 and quickly know and understand the situation in the monitoring area Z.

[0043] FIG. 8 is a system block diagram illustrating an intelligent application image recognition program according to one embodiment of the present invention. In the example of FIG. 8, the intelligent application image recognition program 40 is an AI application program generated by generative AI (GenAI). The intelligent application image recognition program 40 includes an image recognition model database 41, which includes a plurality of object identification feature data 411 and a plurality of object action recognition feature data 412. Furthermore, the image recognition model database 41 analyzes, for example, entities in a video segment that do not perfectly match the object identification feature or object action recognition feature (e.g., vehicle types that correspond to the object identification feature of a non-predefined vehicle, such as a military transport vehicle) based on the input training data. Images of the military transport vehicle are used as training data, and new object identification feature data and object action recognition feature data corresponding to the military transport vehicle are generated based on the training data. Thus, through these training and learning processes, new object identification feature data 411 and new object action recognition feature data 412 are added to the image recognition model database 41.

[0044] Compared with the prior art, the surveillance system capable of automatically generating text summaries of the present invention captures video segments when an alert condition is triggered, and generates a narrative passage and a solitaire summary corresponding to the video segment. Users can obtain the solitaire summary through a portable device and quickly understand the status of the monitored area from the information content of the solitaire summary, eliminating the need to spend a lot of time and effort watching surveillance footage.

[0045] Although the embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and design changes and the like are also included within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0046] 1. A monitoring system that can automatically generate text summaries 1' A surveillance system that can automatically generate text summaries 10. Computer Devices 20 Image Capture Device 30 Signal transmission device 40 Intelligent Application Image Recognition Program 41 Image Recognition Model Database 411 Object Identification Feature Data 412 Object Action Recognition Feature Data 50 cloud servers 60 Portable Electronic Devices C vehicle CP communication protocol D dog h user VS1 Video Segment VS2 Video Segment VS3 Video Segment VS4 Video Segment VS5 Video Segment P1 Postman P2 Electric power company personnel P3 Female visitor P4 Toddlers RA Solitaire Overview S100~S107 steps S1001~S1003 steps S1041~S1042 Steps S1051~S1053 Steps S1061~S1062 Steps Z Monitoring Area

Claims

1. A monitoring system capable of automatically generating a solitaire synopsis including a combination of narrative sentences in a passage that are combined in chronological order, the monitoring system comprising: A portable electronic device; an image capture device for capturing an image of the monitored area; A computing device storing an intelligent application image recognition program, the computing device comprising: (a) setting the scope of the monitoring area and notification conditions; (b) operating the image capture device to capture the image of the monitored area; (c) determining whether the image being captured triggers the notification condition; (d) extracting video segments from the images that meet the notification criteria; (e) dividing the video segment into a series of image files; (f) extracting from each of the image files an entity and an action of the entity that meets the notification condition; (g) generating a passage of narrative text corresponding to each of the subjects and each of the actions of the series of image files, and forming the solitaire synopsis including a combination of the passages of narrative text obtained by combining the passages of narrative text corresponding to the video segments in a chronological order; a computer device that executes the intelligent application image recognition program, a signal transmission device for establishing a communication connection between any one of the portable electronic device, the image capture device, and the computing device; the signal transmission device transmits the solitaire summary from the computing device to the portable electronic device over the communication connection; Surveillance system.

2. The surveillance system of claim 1 , wherein in step (a), the notification conditions include at least one object identification feature and at least one object motion recognition feature.

3. In step (f), the computing device: (f1) extracting, based on at least one of the object identifying features, the entities that match at least one of the object identifying features; (f2) extracting the motion matching at least one of the object motion recognition features based on at least one of the object motion recognition features.

4. The surveillance system of claim 3 , wherein in step (f), the computing device further performs the step of sequentially adding a timestamp to each of the image files.

5. In step (g), the computing device: (g1) generating a subject predicate sentence that matches the subject, and generating an action predicate sentence that matches the action; The monitoring system of claim 3, further comprising: (g2) forming the passage of narrative sentences by combining the subject narrative sentences and the action narrative sentences in chronological order.

6. In step (e), the computing device: (e1) dividing the video segment into the series of image files; The surveillance system of claim 1 , further comprising: (e2) sequentially adding a timestamp to each of the image files.

7. The monitoring system of claim 1 , wherein the computing device is a home computer or a cloud server.

8. 2. The surveillance system of claim 1, wherein the intelligent application image recognition program includes an image recognition model database, the image recognition model database including a plurality of object identification feature data and a plurality of object action recognition feature data, and the plurality of object identification feature data and the plurality of object action recognition feature data are newly created based on training data input into the video segment.

9. 1. A method performed by a surveillance system capable of automatically generating a solitaire summary comprising a combination of narrative passages generated by utilizing image capture and combined in a chronological order to analyze images of a surveillance area captured by an image capture device, the surveillance system comprising: a computing device storing an intelligent application image recognition program, the computing device being configured to execute the intelligent application image recognition program to perform the method; The method comprises: (a) setting the scope of the monitoring area and notification conditions; (b) operating the image capture device to capture the image of the monitored area; (c) determining whether the image being captured triggers the notification condition; (d) extracting video segments from the images that meet the notification criteria; (e) dividing the video segment into a series of image files; (f) extracting from each of the image files an entity and an action of the entity that meets the notification condition; (g) generating a passage of narrative text corresponding to each of the subjects and each of the actions of the series of image files, and forming the solitaire synopsis including a combination of the passages of narrative text obtained by combining the passages of narrative text corresponding to the video segments in a chronological order; (h) transmitting the solitaire summary to a portable electronic device; method.

10. The method of claim 9 , wherein in step (a), the notification condition includes at least one object identification feature and at least one object action recognition feature.

11. In step (f), (f1) extracting the entities that match the at least one object identifying feature based on the at least one object identifying feature; The method of claim 10 , further comprising: (f2) extracting, based on the at least one object action recognition feature, the action that matches the at least one object action recognition feature.

12. The method of claim 11 , further comprising the step of: in step (f), sequentially adding a timestamp to each of the image files.

13. In step (g), (g1) generating a subject predicate sentence that matches the subject, and generating an action predicate sentence that matches the action; The method of claim 11 , further comprising: (g2) forming the passage of narrative sentences by combining the subject narrative sentences and the action narrative sentences in chronological order.

14. In step (e), (e1) dividing the video segment into the series of image files; The method of claim 9 , further comprising: (e2) sequentially adding a timestamp to each of the image files.

Images