System and method for identifying events in video stream
By combining simpler and more comprehensive video analysis algorithms to initially process and verify events in the video stream, the problem of difficult to balance computing costs and accuracy in the prior art is solved, and more efficient and accurate event recognition is achieved.
Patent Information
- Application Number
- CN202411243118.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-09-29
- Filing Date
- 2024-09-05
- Publication Date
- 2025-04-01
AI Technical Summary
When performing event recognition in video streams, prior art is difficult to balance the computational cost and accuracy of video processing algorithms, resulting in simpler algorithms being less accurate, while more comprehensive and more computationally intensive algorithms being more accurate but more computationally costly.
A combination of two video analysis algorithms is adopted: the first video analysis algorithm is used to initially process the video stream, identify candidate events and their types; the second video analysis algorithm (more computational density) is used to verify the benchmark truth of the candidate events and match the results of the first video analysis algorithm. Report events when matched; otherwise, they are not reported and can be used to train the first video analysis model.
It improves the accuracy of event recognition in video streams, while reducing computational costs, avoiding the inaccuracy of simple algorithms and the waste of resources for algorithms with high computational intensive calculations.
Smart Images
Figure CN119741508A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to methods and systems for performing event recognition in video streams, and more particularly to methods and systems for improving the accuracy of performing event recognition in video streams. Background Art
[0002] A variety of video processing algorithms are known and used. Some video processing algorithms are simpler and require relatively less computing power to run, while other video processing algorithms are more comprehensive and require relatively more computing power to run. In addition to the trade-off between speed and computing requirements, there may also be a trade-off in accuracy. A simpler algorithm that may require less computing power may be less accurate, while a more comprehensive algorithm that requires more computing power may be more accurate. What is desired is a method and system for balancing the benefits and costs of various video processing algorithms in detecting events in a video stream. Summary of the invention
[0003] The present disclosure relates to methods and systems for performing event recognition in a video stream, and more specifically to methods and systems for improving the accuracy of performing event recognition in a video stream. An example may be found in a method for identifying an event in a video stream. The illustrative method includes processing a video stream using a first video analysis algorithm (e.g., a simpler less computationally intensive video analysis algorithm) to identify candidate events in the video stream and candidate event types of the candidate events. One or more images representing the candidate events are extracted from the video stream. One or more images representing the candidate events extracted from the video stream are processed using a second video analysis algorithm (e.g., a more comprehensive and more computationally intensive video analysis algorithm) to identify a baseline truth for the candidate event. Determine whether the baseline truth for the candidate event matches a candidate event with a candidate event type determined by the first video analysis algorithm. When the baseline truth for the candidate event matches a candidate event with a candidate event type determined by the first video analysis algorithm, the candidate event with the candidate event type is reported to a monitoring station. When the baseline truth for the candidate event does not match a candidate event with a candidate event type determined by the first video analysis algorithm, the candidate event with the candidate event type is not reported to the monitoring station. In some cases, when the reference truth of a candidate event does not match a candidate event of the candidate event type determined by the first video analysis algorithm, the reference truth of the candidate event determined by the second video analysis algorithm may be used to train the first video analysis model.
[0004] Another example may exist in a system. The exemplary system includes one or more memories for storing a first video analysis algorithm and a second video analysis algorithm, wherein the second video analysis algorithm is more computationally intensive than the first video analysis algorithm. The exemplary system includes one or more controllers operatively coupled to the one or more memories. The one or more controllers are configured to receive a video stream and process the video stream using a first video analysis algorithm to identify candidate events in the video stream and candidate event types of the candidate events. The one or more controllers are configured to extract one or more images representing the candidate event from the video stream, and process the one or more images representing the candidate event extracted from the video stream using a second video analysis algorithm to identify the baseline truth of the candidate event. The one or more controllers are configured to determine whether the baseline truth of the candidate event matches the candidate event with the candidate event type determined by the first video analysis algorithm. The one or more controllers are configured to report the candidate event with the candidate event type when the baseline truth of the candidate event matches the candidate event with the candidate event type determined by the first video analysis algorithm, and not report the candidate event with the candidate event type when the baseline truth of the candidate event does not match the candidate event with the candidate event type determined by the first video analysis algorithm.
[0005] Another example may exist in a method for identifying an event in a video stream. The method includes identifying an event in a video stream using a first video analysis algorithm, and verifying the identified event in the video stream using a second video analysis algorithm, wherein the second video analysis algorithm is more comprehensive than the first video analysis algorithm. The method includes reporting an event identified by the first video analysis algorithm when the identified event is verified by the second video analysis algorithm, and not reporting an event identified by the first video analysis algorithm when the identified event is not verified by the second video analysis algorithm.
[0006] The above summary is provided to facilitate understanding of some innovative features unique to the present disclosure, but is not intended to be a complete description. A comprehensive understanding of the present disclosure can be obtained by viewing the entire specification, claims, drawings and abstract as a whole. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] The present disclosure may be more fully understood by considering the following description of various examples in conjunction with the accompanying drawings, in which:
[0008] Figure 1A is a schematic block diagram illustrating an exemplary system for identifying events in a video stream;
[0009] Figure 1B is a diagram showing the distribution of Figure 1A A schematic block diagram of an exemplary system of;
[0010] Figure 2 is a flow chart illustrating a series of steps that a controller that performs part of the exemplary system of FIG. 1 may be configured to perform;
[0011] Figure 3 is a flow chart illustrating an exemplary method for identifying events in a video stream; and
[0012] Figure 4 is a flow chart illustrating an illustrative method for identifying events in a video stream.
[0013] Although the present disclosure is subject to various modifications and alternative forms, the details thereof have been shown by way of example in the accompanying drawings and will be described in detail. However, it should be understood that it is not intended to limit the present disclosure to the specific examples described. On the contrary, it is intended to cover all modifications, equivalents and alternatives that fall within the essence and scope of the present disclosure. DETAILED DESCRIPTION
[0014] The following description should be read with reference to the accompanying drawings, in which similar elements in different drawings are numbered in the same manner. The accompanying drawings are not necessarily drawn to scale, and they depict examples that are not intended to limit the scope of the present disclosure. Although examples of various elements are shown, those skilled in the art will recognize that many of the examples provided have suitable alternatives that can be utilized.
[0015] It is assumed herein that all numbers are modified by the term "about" unless the content clearly dictates otherwise. The recitation of numerical ranges by endpoints includes all numbers subsumed within that range (eg, 1 to 5 includes 1, 1.5, 2, 2.75, 3, 3.80, 4, and 5).
[0016] As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the content clearly dictates otherwise. As used in this specification and the appended claims, the term "or" is generally employed in its sense including "and / or" unless the content clearly dictates otherwise.
[0017] It should be noted that references to "one embodiment", "some embodiments", "other embodiments", etc. in the specification indicate that the described embodiments may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, these phrases do not necessarily refer to the same embodiment. In addition, when a particular feature, structure, or characteristic is described in conjunction with one embodiment, it is contemplated that the feature, structure, or characteristic is described in conjunction with one embodiment, and it is contemplated that the feature, structure, or characteristic may be applied to other embodiments regardless of whether explicitly described, unless otherwise clearly stated.
[0018] Figure 1A1 is a schematic block diagram showing an exemplary system 10 for identifying events in a video stream 12. Although a single video stream 12 is shown, it should be understood that the system 10 may receive multiple video streams, for example, from each of multiple cameras or video storage devices. The exemplary system 10 includes one or more memories, generally shown at 14, including, for example, a memory 14a, a memory 14b, and a memory 14c. Although three memories 14 are shown, this is merely exemplary. The one or more memories 14 are collectively configured to store a first video analysis algorithm 16 and a second video analysis algorithm 18. One or more controllers 20, labeled as controllers 20a, 20b, and 20c, are operably coupled to the one or more memories 14. A monitoring station 26 is operably coupled to one or more of the controllers 20. The monitoring station 26 may be configured to allow a user to view, for example, images from the video stream 12, and to view possible event alerts triggered by one or more controllers 20. For example, in some cases, the system 10 may reside within a single computing device. In other cases, the system 10 may be distributed across multiple computing devices.
[0019] Figure 1B is a schematic block diagram showing an exemplary system 10 distributed across multiple computing devices. Figure 1B In the example shown, the exemplary system 10 is distributed across edge devices 24 and servers 22 (e.g., cloud servers or other servers, including locally deployed servers). Gateway 29 provides a communication path between edge devices 24 and server 22. In some cases, server 22 may serve one or more other edge devices, such as edge device 27, via gateway 29. Edge devices 24 and 27 may have reduced computing capabilities relative to server 22.
[0020] exist Figure 1B middle, Figure 1A The memory 14a and controller 20a are provided in the edge device 24, and the memory 14b and controller 20b are provided in the server 22 (e.g., a cloud server). When so provided, the edge device 24 stores and executes the first video analysis algorithm 16, and the server 22 stores and executes the second video analysis algorithm 18. In some cases, the edge device 24 may execute the first video analysis algorithm 16 on the video stream 12 to determine candidate events and candidate event types. The edge device 24 may extract one or more images representing the candidate event from the video stream 12. The one or more images may include a single image, two or more images, and / or video clips. The one or more images extracted from the video stream 12 may be sent to the server 22 via the gateway 29. The server 22 may execute the second video analysis algorithm 18 on the one or more images extracted from the video stream 12 by the edge device 24 to determine the baseline truth, and confirm or reject what the first video analysis algorithm 16 of the edge device 24 found.
[0021] The server may interact with the monitoring station 26. The server may report a candidate event having a candidate event type to the monitoring station 26 when the baseline truth of the candidate event matches a candidate event having the candidate event type determined by the first video analytics algorithm 16 of the edge device 24. When the baseline truth of the candidate event does not match a candidate event having the candidate event type determined by the first video analytics algorithm 16 of the edge device 24, the candidate event having the candidate event type is not reported to the monitoring station 26. In some cases, when the baseline truth of the candidate event does not match a candidate event having the candidate event type determined by the first video analytics algorithm 16 of the edge device 24, the first video analytics algorithm 16 may be trained using the baseline truth of the candidate event determined by the second video analytics algorithm 18 of the server 22.
[0022] To give a few examples of how the baseline truth provided by the second video analysis algorithm 18 can be used for event verification, a traffic cone may be identified by the first video analysis algorithm 16 as a "wandering person", while the second video analysis algorithm 18 may correctly identify the traffic cone as a "red and white traffic cone". In another example, a computer may be identified by the first video analysis algorithm 16 as a "wandering person", while the second video analysis algorithm 18 may correctly identify a "small monitor". In another example, a floor may be identified by the first video analysis algorithm 16 as an "object left behind" or a "discarded object", while the second video analysis algorithm model 18 may correctly identify a "patch in the ground". In yet another example, a chair may be identified by the first video analysis algorithm 16 as an "object left behind" or a "discarded object", while the second video analysis algorithm 18 may correctly identify a "black office chair located in front of a desk". These are just examples.
[0023] To give an example of conditional alarm suppression, the first video analysis algorithm 16 may trigger a cross-line alarm in response to seeing a person crossing a safety line drawn near a platform at a train station. The second video analysis algorithm 18 may identify this scenario as "a person actively boards a train parked at a train platform." Since a person needs to cross the safety line adjacent to the train platform to actively board the train, the cross-line alarm triggered by the first video analysis algorithm may be suppressed and not sent to the monitoring station. This may eliminate some false alarms, thereby not unnecessarily distracting the operator of the monitoring station. However, if the second video analysis algorithm 18 identifies the scenario as "a person crosses the safety line when a train is approaching the train platform," the cross-line alarm triggered by the first video analysis algorithm may not be suppressed and may be forwarded to the monitoring station.
[0024] To give a few examples of anti-spoofing, the first video analysis algorithm 16 may raise a fire alarm in response to detecting a fire within a scene. The second video analysis algorithm 18 may correctly identify that the scene includes a painting on the wall that includes a fire scene, or perhaps a large screen television playing a show of a fire burning. Therefore, the fire alarm generated by the first video analysis algorithm 16 will be suppressed because the system determines that there is no actual fire occurring. As another example, a person who is not authorized to enter through a particular door may attempt to spoof an access control camera by holding up a badge that includes a picture of the authorized person's face. Even though the first video analysis algorithm 16 may detect the picture of the person and identify it as actually that person, the second video analysis algorithm 18 may determine that it is not a person's animate face, but is merely a picture of a person.
[0025] Figure 2 is a flow chart showing an exemplary series of steps 28 that each of the one or more controllers 20 may be configured to perform. Some of the series of steps 28 may be performed by the controller 20a in the edge device 24. Some of the series of steps 28 may be performed by the controller 20b in the server 22. The one or more controllers 20 may be configured to receive the video stream 12, as shown in box 30. The one or more controllers 20 may be configured to process the video stream 12 using the first video analysis algorithm 16 to identify candidate events in the video stream 12 and candidate event types of the candidate events, as shown in box 32. The one or more controllers 20 may be configured to extract one or more images representing the candidate event from the video stream, as shown in box 34. The one or more controllers 20 may be configured to process the one or more images representing the candidate event extracted from the video stream 12 using the second video analysis algorithm 18 to identify a ground truth of the candidate event, as shown in box 36.
[0026] The one or more controllers 20 may be configured to determine whether the baseline truth of the candidate event matches a candidate event of the candidate event type determined by the first video analysis algorithm 16, as shown at box 38. The one or more controllers 20 may be configured to report the candidate event of the candidate event type when the baseline truth of the candidate event matches the candidate event of the candidate event type determined by the first video analysis algorithm 16, as shown at box 40. The one or more controllers 20 may be configured to not report the candidate event of the candidate event type when the baseline truth of the candidate event does not match the candidate event of the candidate event type determined by the first video analysis algorithm 16. In some cases, the one or more controllers 20 may be configured to report the baseline truth of the candidate event when the baseline truth of the candidate event does not match the candidate event of the candidate event type.
[0027] In some cases, an edge device (such as edge device 24) may be configured to initially process video stream 12 using first video analysis algorithm 16 to identify candidate events in the video stream and candidate event types of the candidate events, and a server (such as server 22) may be configured to process one or more images extracted from video stream 12 representing the candidate events to identify a ground truth of the candidate event using second video analysis algorithm 18. In some cases, first video analysis algorithm 16 may be trained using one or more images extracted from video stream 12 representing the candidate events and the corresponding ground truth of the candidate events.
[0028] Figure 3 4 is a flow chart showing an exemplary method 44 for identifying events in a video stream, such as video stream 12. The exemplary method 44 includes processing the video stream using a first video analysis model, such as first video analysis model 16, to identify candidate events in the video stream and candidate event types for the candidate events, as shown in block 46. In some cases, the first video analysis algorithm may include one or more object detection algorithms, object tracking algorithms, or object classification algorithms. As an example, the one or more object detection and / or object tracking algorithms may include one or more of an object detection algorithm, an intrusion detection algorithm, a loitering detection algorithm, a discarded object detection algorithm, and an object tracking algorithm. In some cases, the first video analysis algorithm may identify candidate events and candidate event types in the video stream based at least in part on outputs from one or more of the one or more object detection and / or tracking algorithms of the first video analysis algorithm.
[0029] One or more images representing the candidate event are extracted from the video stream, as shown in block 48. In some cases, the one or more images representing the candidate event from the video stream may include cropped images representing the candidate event from a video frame of the video stream. In some cases, the one or more images representing the candidate event from the video stream may include a video clip. The one or more images representing the candidate event extracted from the video stream are processed using a second video analysis algorithm (such as the second video analysis algorithm 18) to identify the baseline truth of the candidate event, as shown in block 50. In some cases, the second video analysis algorithm may be a more comprehensive video analysis algorithm than the first video analysis algorithm. In some cases, the second video analysis algorithm may be more computationally intensive than the first video analysis algorithm. In some cases, the first video analysis algorithm may be executed by an edge device (such as edge device 24), and the second video analysis algorithm may be executed by a server (such as server 22). For example, edge device 24 may process the video stream using the first video analysis algorithm to identify the candidate event and the candidate event type of the candidate event in the video stream, and server 22 may process the one or more images representing the candidate event extracted from the video stream using the second video analysis model to identify the baseline truth of the candidate event.
[0030] A determination is made as to whether the base truth of the candidate event matches a candidate event having a candidate event type determined by the first video analytics algorithm, as shown in block 52. In some cases, server 22 may determine whether the base truth of the candidate event matches a candidate event having the candidate event type. When the base truth of the candidate event matches a candidate event having the candidate event type determined by the first video analytics algorithm, the candidate event having the candidate event type is reported to a monitoring station (such as monitoring station 26), as shown in block 54. When the base truth of the candidate event does not match a candidate event having the candidate event type determined by the first video analytics algorithm, the candidate event having the candidate event type is not reported to the monitoring station, as shown in block 56.
[0031] In some cases, the method 44 may include reporting the reference truth of the candidate event to the monitoring station when the reference truth of the candidate event does not match a candidate event of the candidate event type, as shown in block 58. In some cases, the method 44 may include training the first video analysis algorithm using one or more images extracted from the video stream representing the candidate event and the corresponding reference truth of the candidate event, as shown in block 60.
[0032] In some cases, the second video analysis algorithm may include a visual transformer large language model (ViTLLM) that outputs a text description as a base truth for the candidate event. In some cases, the first video analysis algorithm outputs a text description of the candidate event and a candidate event type for the candidate event, and determining whether the base truth for the candidate event matches a candidate event with the candidate event type determined by the first video analysis algorithm may include comparing the text description of the candidate event and the candidate event type for the candidate event output by the first video analysis algorithm with a text description of the base truth for the candidate event output by the ViTLLM model. In some cases, comparing the text description of the candidate event and the candidate event type for the candidate event output by the first video analysis algorithm with a text description of the base truth for the candidate event output by the ViTLLM model may include detecting one or more keywords and / or keyword synonyms in the text description of the candidate event, the candidate event type, and / or in the text description of the base truth for the candidate event.
[0033] Figure 46 is a flow chart illustrating an exemplary method 62 for identifying an event in a video stream. The method 62 includes identifying an event in a video stream using a first video analysis algorithm, as indicated by block 64. Once the event is detected by the first video analysis algorithm, the identified event is verified using a second video analysis algorithm that is more comprehensive than the first video analysis algorithm, as indicated by block 66. When the identified event is verified by the second video analysis algorithm, the event identified by the first video analysis algorithm is reported, as indicated by block 68. When the identified event is not verified by the second video analysis algorithm, the event identified by the first video analysis algorithm is not reported, as indicated by block 70.
[0034] Although several exemplary embodiments of the present disclosure have been described in this manner, it will be readily understood by those skilled in the art that other embodiments may be made and used within the scope of the claims appended hereto. However, it should be understood that the present disclosure is in many respects merely illustrative. Details (especially details related to shape, size, arrangement of parts, and exclusion and order of steps) may be changed without exceeding the scope of the present disclosure. Of course, the scope of the present disclosure is defined in the language of the appended claims.
Claims
1. A method for identifying an event in a video stream, the method comprising: Processing the video stream using a first video analysis algorithm to identify candidate events in the video stream and candidate event types of the candidate events; extracting one or more images representing the candidate event from the video stream; processing the one or more images extracted from the video stream and representing the candidate event using a second video analysis algorithm to identify a baseline truth for the candidate event; determining whether the reference reality of the candidate event matches the candidate event of the candidate event type determined by the first video analysis algorithm; When the reference truth of the candidate event matches the candidate event of the candidate event type determined by the first video analysis algorithm, reporting the candidate event of the candidate event type to a monitoring station; as well as When the reference truth of the candidate event does not match the candidate event of the candidate event type determined by the first video analysis algorithm, the candidate event of the candidate event type is not reported to the monitoring station.
2. The method according to claim 1, further comprising: When the reference truth of the candidate event does not match the candidate event having the candidate event type, the reference truth of the candidate event is reported to the monitoring station.
3. The method of claim 1, wherein the second video analysis algorithm is a more comprehensive video analysis model than the first video analysis algorithm.
4. The method of claim 1, wherein the second video analysis algorithm is more computationally intensive than the first video analysis algorithm.
5. The method according to claim 4, wherein: The edge device processes the video stream using the first video analysis algorithm to identify the candidate events in the video stream and the candidate event types of the candidate events; and The server processes the one or more images representing the candidate event extracted from the video stream using the second video analysis algorithm to identify the reference truth of the candidate event. The method of claim 5 , wherein the server determines whether the reference truth of the candidate event matches the candidate event of the candidate event type.
7. The method of claim 1, wherein the one or more images from the video stream representing the candidate event comprise cropped images from video frames of the video stream representing the candidate event.
8. The method of claim 1, wherein the one or more images from the video stream representing the candidate event comprise a video clip.
9. The method of claim 1, wherein the first video analysis algorithm comprises one or more object detection algorithms, object tracking algorithms, or object classification algorithms.
10. A system, comprising: one or more memories for storing a first video analytics algorithm and a second video analytics algorithm, wherein the second video analytics algorithm is more computationally intensive than the first video analytics algorithm; One or more controllers operatively coupled to the one or more memories, the one or more controllers being configured to: Receive video stream; Processing the video stream using the first video analysis algorithm to identify candidate events in the video stream and candidate event types of the candidate events; extracting one or more images representing the candidate event from the video stream; processing the one or more images extracted from the video stream and representing the candidate event using the second video analysis algorithm to identify a baseline truth for the candidate event; determining whether the reference reality of the candidate event matches the candidate event of the candidate event type determined by the first video analysis algorithm; reporting the candidate event of the candidate event type when the reference truth of the candidate event matches the candidate event of the candidate event type determined by the first video analysis algorithm; as well as When the reference truth for the candidate event does not match the candidate event of the candidate event type determined by the first video analytics algorithm, the candidate event of the candidate event type is not reported.