Display method, camera, edge computing device, and monitoring system

Abstract

The application provides a display method, a camera, an edge computing device and a monitoring system. The method comprises: acquiring monitoring video data; acquiring a first detection instruction; and in response to the first detection instruction, displaying description information of a first target object in an image included in the monitoring video data, the first target object being one or more. That is, after the first detection instruction is acquired, the monitoring video data is automatically detected to detect the first target object existing in the image in the monitoring video data, and the description information of the first target object is displayed. In this way, the monitoring video data does not need to be manually checked frame by frame, the time for manually checking frame by frame can be greatly reduced, and the video information screening efficiency is improved. In addition, one or more first target objects can be detected at a time, and the detection efficiency is high. Furthermore, the description information of the first target object is displayed, so that a user can quickly master specific conditions appearing in the monitoring video data.

Description

Technical Field

[0001] This application relates to the media field, and more particularly to a display method, camera, edge computing device, and monitoring system. Background Technology

[0002] Currently, cameras are being installed in an increasing number of places for anomaly monitoring. For example, shopping malls install cameras at various entrances and exits to monitor people with unusual behavior or appearance. Cameras are also installed on roads to monitor speeding, illegally parked, and other violations. Furthermore, cameras are installed in homes to monitor children or the elderly who may be in danger.

[0003] Existing anomaly monitoring technologies mainly employ two methods. One method involves manually reviewing surveillance video frame by frame: users need to examine the surveillance video data captured by the camera frame by frame for anomaly monitoring. This method is time-consuming, and manual processing can easily miss crucial details, reducing the accuracy and reliability of anomaly location. The other method is image search-based anomaly monitoring: users upload an image, and the computer searches for anomalies in the surveillance video data captured by the camera based on that image. While this method reduces manual time consumption, it can only monitor one type of anomaly at a time. When monitoring multiple anomalies, users need to upload images multiple times, and the computer needs to process them multiple times, resulting in low efficiency.

[0004] In summary, existing technologies for anomaly monitoring suffer from problems such as long processing time, low efficiency, and insufficient accuracy, and there is an urgent need for more efficient and intelligent monitoring methods to improve the effectiveness of anomaly monitoring. Summary of the Invention

[0005] In view of this, this application provides a display method, a camera, an edge computing device, and a monitoring system. This method can improve the efficiency, reliability, and effectiveness of monitoring in specific situations.

[0006] In a first aspect, this application provides a display method that can be applied to the field of surveillance. The method includes: acquiring surveillance video data, the surveillance video data including multiple frames of images; acquiring a first detection command; and in response to the first detection command, displaying descriptive information of a first target object in the images included in the surveillance video data, wherein the first target object is one or more.

[0007] The first target object may include objects with specific behaviors and / or objects with specific appearances, and can also be referred to as specific objects. The specific behaviors and specific appearances can be set according to requirements; for example, specific behaviors may include abnormal behaviors, and specific appearances may include abnormal appearances; this application does not impose any limitations on this. In other words, after receiving the first detection instruction, the electronic device (such as a camera, edge computing device, server, etc.) of this application can automatically detect the surveillance video data to identify objects with specific behaviors and / or specific appearances (i.e., the first target object) in the images of the surveillance video data and display the descriptive information of the first target object. This eliminates the need for manual frame-by-frame review of the surveillance video data, significantly reducing the time spent on manual frame-by-frame searching and thus improving the efficiency of video information filtering; it also reduces the risk of information omission, resulting in high reliability and effectiveness. Furthermore, this application can detect one or more first target objects and / or one or more specific behaviors at once, resulting in high detection efficiency; and the display of the descriptive information of the specific object allows users (such as monitoring / inspection personnel) to quickly grasp the specific situation appearing in the surveillance video data.

[0008] For example, the first detection instruction can be used to instruct the edge computing device to detect specific objects in the images included in the surveillance video data, and generate and display descriptive information of the specific objects. Specific objects may include specific people or specific items (such as specific animals, specific plants, specific vehicles, specific machines, etc.); for example, people wearing masks and hats, people holding controlled knives, illegally parked vehicles, speeding vehicles, people fighting, people illegally trespassing, lakes with water levels above the warning line, vegetation with a growth rate exceeding a threshold, machines operating at abnormal speeds, etc.

[0009] For example, the first detection command can be triggered by the user or the system.

[0010] For example, the display method can be executed by any one of the electronic devices, such as a camera, an edge computing device, or a server, or it can be executed collaboratively by any two or more electronic devices, such as a camera, an edge computing device, or a server. This application does not limit this.

[0011] For example, the acquired surveillance video data can be one or more channels.

[0012] For example, the monitoring fields covered by this application include, but are not limited to: public safety monitoring (such as commercial area monitoring), traffic monitoring, environmental monitoring, health monitoring (such as home monitoring), and industrial monitoring. It should be understood that the definition of a specific object may differ across different monitoring fields or different application scenarios within the same monitoring field.

[0013] For example, descriptive information of a first target object in an image included in one or more surveillance video streams can be displayed.

[0014] According to the first aspect, the descriptive information of the first target object includes information describing the appearance of the first target object and / or the behavior of the first target object. Thus, based on the displayed descriptive information of the first target object, users (such as monitoring / inspection personnel) can quickly grasp the appearance and / or behavior of a specific object.

[0015] For example, the information used to describe the appearance of the first target object can be called the appearance description information of the first target object; the information used to describe the behavior of the first target object can be called the behavior description information of the first target object.

[0016] It should be understood that, when the first target object is an object with a specific appearance, the descriptive information of the first target object includes information describing the appearance of the first target object. When the first target object is an object with a specific behavior, the descriptive information of the first target object includes information describing the behavior of the first target object; alternatively, the descriptive information of the first target object includes information describing the behavior of the first target object and information describing the appearance of the first target object (in this case, the descriptive information of the first target object may also be referred to as the descriptive information of a specific event (hereinafter referred to as the first target event)). When the first target object is an object with both a specific appearance and specific behavior, the descriptive information of the first target object includes information describing the behavior of the first target object and information describing the appearance of the first target object (in this case, the descriptive information of the first target object may also be referred to as the descriptive information of the first target event).

[0017] For example, artificial intelligence (AI) algorithms (such as AI models) can be used to generate descriptive information about the first target object.

[0018] According to the first aspect, or any implementation of the first aspect above, the description information of the first target object includes at least one of the following: identification information, attribute information, statistical information, feature information, thumbnail, or time information.

[0019] For example, the identification information may include an object identifier of the first target object and / or a behavior identifier of the first target object. The object identifier can be used to uniquely identify an object; for example, Person 1 identifies user 1, Person 2 identifies user 2, Vehicle 1 identifies vehicle 1, Vehicle 2 identifies vehicle 2, etc. The behavior identifier can be used to uniquely identify a behavior; for example, Behavior 1 identifies fighting, Behavior 2 identifies speeding, Behavior 3 identifies damaging a trash can, Behavior 4 identifies trespassing, Behavior 5 identifies loitering, etc.

[0020] For example, attribute information can be used to describe the attributes of a first target object, which can be a person or an item.

[0021] For example, the feature information may include information describing the appearance features of the first target object (which may be referred to as appearance characteristics) and / or information describing the behavioral features of the first target object (which may be referred to as behavioral features). For example, information describing the appearance features of the first target object may include "mask", "hat", "green dress", "white high heels", etc. As another example, information describing the behavioral features of the first target object may include "holding a stick", "damaging a trash can", "illegally entering", "loitering", etc.

[0022] For example, time information can refer to the timestamp of the image of the first target object appearing in the surveillance video data.

[0023] For example, a thumbnail may refer to a thumbnail of the image in the surveillance video data in which the first target object appears.

[0024] For example, the statistical information may include statistical information of the first target object (such as the frequency and duration of the first target object appearing in the surveillance video data), and statistical information of specific behaviors of the first target object (such as the frequency and duration of specific behaviors).

[0025] For example, the appearance description information may include at least one of the following: the object identifier of the first target object, the attributes of the first target object, the statistical information of the first target object, the appearance features of the first target object, the thumbnail of the first target object, or the timestamp of the image in which the first target object is located.

[0026] For example, the behavior description information may include at least one of the following: the object identifier of the first target object, the behavior identifier of the first target object, the attribute of the first target object, the statistical information of the specific behavior of the first target object, the behavior characteristics of the first target object, the thumbnail of the first target object, or the timestamp of the image where the first target object is located.

[0027] For example, the edge computing device can display multiple descriptive information on the interactive interface. A descriptive information can be the appearance description of a first target object, or the behavior description of a first target object, or the appearance description of a first target object plus the behavior description (i.e., the description of a first target event).

[0028] For example, a description (i.e., the appearance description of a first target object) is: [10:03:55~10:30:20][Person][Person 1 mask hat green dress white high heels].

[0029] For example, a description (i.e., the appearance description of a first target object) is: [00:15:25~00:30:20][Iitem][Vehicle 1 Green sports car license plate****].

[0030] For example, a descriptive message (i.e., a behavioral description of a first target object) is: [10:03:55~10:30:20][Person 3Behavior 1 Holding a stick].

[0031] For example, a descriptive message (i.e., a behavioral description of a first target object) is: [00:15:25~00:30:20][Person 10Behavior 2 speed 100km / h].

[0032] For example, a descriptive message (i.e., a behavioral description of a first target event) is: [12:22:01~13:45:30][Person][Person3Behavior 1 Mask, Hat, Black Jacket, Basketball Shoes, Holding a Stick].

[0033] According to the first aspect, or any implementation of the first aspect above, the method further includes: in response to the first detection instruction, displaying a video summary of the surveillance video data. The video summary of the surveillance video data can be generated based on the description information of the first target object; this helps users (such as monitoring / inspection personnel) quickly obtain key content from the surveillance video data, improving video filtering efficiency.

[0034] For example, a natural language model can be used to generate video summaries.

[0035] For example, the video summary can be a summary of the overall surveillance video data or a summary of all the first target objects.

[0036] According to the first aspect, or any implementation of the first aspect above, the method further includes: in response to the first detection instruction, displaying a playback option corresponding to the target video segment, wherein the target video segment is composed of multiple frames of images containing the first target object in the monitoring video data. In other words, this application automatically concatenates multiple frames of images containing the first target object to form a time-series trajectory and event chain (i.e., the target video segment), which can help users (such as monitoring / detection personnel) quickly locate the node where the first target object appears, achieving efficient event tracing.

[0037] For example, the description information of the first target object can serve as a summary of the playback options; in this way, users (such as monitoring / inspection personnel) can obtain the key content of each target video segment through the description information of the first target object; and can accurately locate relevant video segments of a specific event based on time information and other information in the description information.

[0038] According to the first aspect, or any implementation of the first aspect above, there are multiple playback options. The method further includes: in response to a playback command, determining a target playback option from the multiple playback options and playing the target video segment corresponding to the target playback option. This facilitates users (such as monitoring / inspection personnel) to quickly view video segments including the first target object.

[0039] In one possible approach, the target video segment can be composed of images containing the first target object stitched together in chronological order; thus, the displayed target video segment is a single-view image.

[0040] In one possible approach, the target video clip can be stitched together from images containing the first target object according to viewpoints; thus, the displayed target video clip is a panoramic image (or a multi-view image).

[0041] According to the first aspect, or any implementation of the first aspect above, the method further includes: acquiring filtering information; acquiring a second detection instruction; and, in response to the second detection instruction, displaying descriptive information of a second target object in the images included in the target video segment, wherein the target video segment consists of multiple frames of images containing the first target object in the monitoring video data, and the filtering information is used to filter the second target object from the first target object. The filtering information can be customized by the user (e.g., monitoring / inspection personnel); thus, the electronic device can automatically filter the second target object from the first target object that matches the user's (e.g., monitoring / inspection personnel's) needs, achieving rapid filtering of specific target objects.

[0042] For example, the second target object may be a subset of the first target object, and the description information of the second target object may be a subset of the description information of the first target object.

[0043] For example, the method further includes: in response to the second detection instruction, displaying a video summary of the surveillance video data; wherein the video summary of the surveillance video data may be a summary of all second target objects.

[0044] According to the first aspect, or any implementation of the first aspect above, the description information is displayed in a preset format.

[0045] For example, the video summary may be displayed in another preset format.

[0046] According to the first aspect, or any of the implementations of the first aspect above, the preset format is either user-defined or system-defined. This allows for the fulfillment of personalized needs by users (such as monitoring / inspection personnel).

[0047] For example, the preset format corresponding to the video summary can also be a user-defined setting or a system setting.

[0048] For example, when the preset format is set by the system, user-defined extensions can also be supported.

[0049] According to the first aspect, or any implementation of the first aspect above, the surveillance video data is multi-channel, and the method further includes: acquiring a selection instruction; in response to the selection instruction, determining target surveillance video data from the multi-channel surveillance video data; in response to the first detection instruction, displaying descriptive information of a first target object in the image included in the surveillance video data, including: in response to the first detection instruction, displaying descriptive information of the first target object in the image included in the target surveillance video data. In this way, specific situations of the surveillance video data desired by the user can be monitored, meeting the user's personalized needs.

[0050] It should be understood that when no selection instruction is received, the monitoring video data targeted by the first detection instruction can be monitoring video data collected by all cameras, or monitoring video data collected by preset cameras.

[0051] According to the first aspect, or any implementation of the first aspect above, the method further includes: obtaining a time setting instruction; determining a detection time period in response to the time setting instruction; and displaying descriptive information of a first target object in the images included in the monitoring video data in response to the first detection instruction, including: displaying descriptive information of the first target object in the images included in the monitoring video data located within the detection time period in response to the first detection instruction. In this way, specific situations of the monitoring video data within a user-expected time period can be monitored, meeting the user's personalized needs.

[0052] It should be understood that when no time setting instruction is received, the time period corresponding to the monitoring video data targeted by the first detection instruction can be the current day, the previous day, the current week, the previous week, a preset time period, or other time periods. This application does not impose any restrictions on this.

[0053] According to the first aspect, or any implementation of the first aspect above, the method further includes: in response to the first detection instruction, displaying the image included in the surveillance video data and highlighting the first target object.

[0054] This allows users to intuitively perceive the key points in the video, enhancing their understanding and trust in the system's functions.

[0055] For example, highlighting, such as highlighting the outline of the first target object, or zooming in on the first target object.

[0056] According to the first aspect, or any implementation of the first aspect above, the method further includes: acquiring other sensor data associated with the surveillance video data, wherein the other sensor data is used to assist in generating the descriptive information. In this way, by combining data from other modalities to determine the first target object in the image included in the surveillance video data and the descriptive information of the first target object, the accuracy of the identified first target object can be improved, as well as the accuracy and comprehensiveness of the generated descriptive information of the first target object.

[0057] For example, other sensor data may include, but are not limited to: audio data, radar data, infrared data, temperature, data collected by vibration sensors, data collected by smoke sensors, etc.

[0058] According to the first aspect, or any implementation of the first aspect above, there are multiple playback options, which are displayed in chronological order. That is, all target video segments corresponding to the first target object are integrated into a timeline for display, making it easy for users to clearly understand the timeline.

[0059] Each first target object can correspond to one or more target video segments, and the multiple frames included in a target video segment are consecutive frames; in this case, a playback option can correspond to a target video segment corresponding to a first target object.

[0060] According to the first aspect, or any implementation of the first aspect above, each target video segment corresponding to the same first target object corresponds to one playback option. In other words, all target video segments corresponding to the same first target object are integrated together, making it convenient for users to view all video segments related to the same first target object.

[0061] In this scenario, one playback option can correspond to all target video segments associated with a primary target object.

[0062] According to the first aspect, or any implementation of the first aspect above, multiple first target objects with the same behavior correspond to a single playback option for their respective target video segments. In other words, all target video segments corresponding to the same specific behavior are integrated together, making it easier for users to view all video segments related to the same specific behavior.

[0063] Secondly, this application provides a monitoring system, which includes a camera and an edge computing device, wherein:

[0064] The camera is used to collect surveillance video data, which includes multiple frames of images.

[0065] The edge computing device is used to acquire the surveillance video data; acquire a first detection command; and, in response to the first detection command, display descriptive information of a first target object in the image included in the surveillance video data.

[0066] According to the second aspect, the camera is one or more, and one camera is used to collect one or more surveillance video data.

[0067] According to the second aspect, or any implementation of the second aspect above, the edge computing device is one or more; when there are multiple edge computing devices, the multiple edge computing devices are used to collaboratively generate descriptive information of the first target object in the image included in the surveillance video data.

[0068] For example, in the case of multiple edge computing devices, one of the edge computing devices may also generate the description information of the first target object in the image included in the surveillance video data.

[0069] It should be understood that the edge computing device of the second aspect and any implementation thereof can also be used to execute the methods of the first aspect and any implementation thereof, which will not be elaborated here.

[0070] For example, this application also provides a monitoring system, which includes a camera, an edge computing device, and a server, wherein:

[0071] The camera is used to collect surveillance video data, which includes multiple frames of images.

[0072] The edge computing device is used to forward the surveillance video data;

[0073] The server is used to acquire the surveillance video data; acquire a first detection command; and, in response to the first detection command, display descriptive information of a first target object in the image included in the surveillance video data.

[0074] For example, there may be one or more servers; in the case of multiple servers, the multiple servers are used to collaboratively generate descriptive information of the first target object in the images included in the surveillance video data.

[0075] For example, in the case of multiple servers, one of the servers may also generate the description information of the first target object in the image included in the surveillance video data.

[0076] It should be understood that this server can also be used to execute the methods in the first aspect and any implementation thereof, which will not be elaborated here.

[0077] Thirdly, this application provides an edge computing device for use with respect to edge computing devices.

[0078] Acquire surveillance video data, which includes multiple frames of images;

[0079] Obtain the first detection command;

[0080] In response to the first detection command, the description information of the first target object in the image included in the surveillance video data is displayed.

[0081] It should be understood that the edge computing device of the third aspect can also be used to execute the methods in the first aspect and any implementation thereof, which will not be elaborated here.

[0082] Fourthly, this application provides a camera for use in...

[0083] Collect surveillance video data, which includes multiple frames of images;

[0084] Obtain the first detection command;

[0085] In response to the first detection command, the description information of the first target object in the image included in the surveillance video data is displayed.

[0086] It should be understood that the camera in the fourth aspect can also be used to execute the methods in the first aspect and any of the implementations of the first aspect, which will not be elaborated here.

[0087] In one possible implementation, one or more edge computing devices and one or more cameras can collaboratively generate descriptive information of the first target object in the image included in the surveillance video data.

[0088] By way of example, this application provides a server for...

[0089] Collect surveillance video data, which includes multiple frames of images;

[0090] Obtain the first detection command;

[0091] In response to the first detection command, the description information of the first target object in the image included in the surveillance video data is displayed.

[0092] It should be understood that this server can also be used to execute the methods in the first aspect and any implementation thereof, which will not be elaborated here.

[0093] In one possible implementation, one or more edge computing devices and one or more servers can collaboratively generate descriptive information of the first target object in the image included in the surveillance video data.

[0094] In one possible implementation, one or more edge computing devices, one or more servers, and one or more cameras can collaboratively generate descriptive information of the first target object in the images included in the surveillance video data.

[0095] Fifthly, this application provides a detection method, which includes: first, acquiring surveillance video data; then, acquiring annotation information of a first target object in the images included in the surveillance video data; and finally, generating descriptive information of the first target object based on the annotation information. In this way, the automatic identification of the first target object and the generation of its descriptive information by electronic devices (such as cameras, edge computing devices, servers, etc.) can significantly reduce the time required for manual frame-by-frame searching, improve the efficiency of video information filtering, and reduce the risk of information omission.

[0096] In one possible implementation, the detection method of this application can be executed by any electronic device, including a camera, an edge computing device, or a server.

[0097] In one possible implementation, the detection method of this application can be executed collaboratively by any two or more electronic devices, including cameras, edge computing devices, or servers. For example, the detection method of this application can be executed collaboratively by one or more edge computing devices and one or more servers. Alternatively, the detection method of this application can be executed collaboratively by one or more edge computing devices, one or more servers, and one or more cameras.

[0098] According to the fifth aspect, obtaining the annotation information of the first target object in the images included in the surveillance video data includes: identifying objects with a specific appearance and their annotation information in the images included in the surveillance video data; and identifying objects with specific behaviors and their annotation information in the images included in the surveillance video data. In this way, objects with a specific appearance and / or specific behaviors (i.e., the first target object) can be automatically identified, significantly reducing the time required for manual frame-by-frame searching of the first target object.

[0099] According to the fifth aspect, or any implementation of the fifth aspect above, based on the annotation information of the first target object, the description information of the first target object is generated, including: concatenating the annotation information of the same first target object to obtain the annotation information set of each first target object; generating statistical information and / or statistical information of specific behaviors of each first target object based on the annotation information set of each first target object; integrating the annotation information set and statistical information of each first target object (including the statistical information of each first target object and / or the statistical information of specific behaviors of each first target object) to obtain the description information of each first target object.

[0100] In this way, multiple associations (temporal association and spatial association) can be made with the same primary target object to generate an event chain of the same primary target object (i.e., the annotation information of the same primary target object), which facilitates event backtracking.

[0101] Sixthly, this application provides a detection device, which includes:

[0102] The data acquisition module is used to acquire surveillance video data.

[0103] The recognition module is used to obtain the annotation information of the first target object in the image included in the surveillance video data.

[0104] The integration module is used to generate description information of the first target object based on the annotation information of the first target object.

[0105] It should be understood that the detection device of the sixth aspect can perform the methods in the fifth aspect or any possible implementation of the fifth aspect described above.

[0106] It should be understood that the detection device can be deployed on any electronic device in a camera, edge computing device, or server; or, distributed across any two or more electronic devices in a camera, edge computing device, or server.

[0107] In a seventh aspect, this application provides an electronic device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, cause the electronic device to perform the method of the first aspect or any possible implementation thereof; or, cause the electronic device to perform the method of the fifth aspect or any possible implementation thereof.

[0108] Eighthly, this application provides a chip including one or more interface circuits and one or more processors; the one or more processors receive or transmit data through the one or more interface circuits, and when the one or more processors execute computer instructions, cause the electronic device to perform the method of the first aspect or any possible implementation of the first aspect; or, cause the electronic device to perform the method of the fifth aspect or any possible implementation of the fifth aspect.

[0109] Ninthly, this application provides a computer-readable storage medium storing a computer program that, when run on a computer or processor, causes the computer or processor to perform the method of the first aspect or any possible implementation thereof; or causes an electronic device to perform the method of the fifth aspect or any possible implementation thereof.

[0110] In a tenth aspect, this application provides a computer program product comprising computer instructions that, when executed by a computer or processor, cause the computer or processor to perform the method of the first aspect or any possible implementation thereof; or cause the computer or processor to perform the method of the fifth aspect or any possible implementation thereof.

[0111] In this embodiment, the monitoring system, camera, edge computing device, electronic device, computer-readable storage medium, computer program product, etc. are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding methods provided above. Attached Figure Description

[0112] Figure 1A This is a schematic diagram illustrating one application scenario of an embodiment of this application;

[0113] Figure 1B This is a schematic diagram illustrating another application scenario of an embodiment of this application;

[0114] Figure 1C This is a schematic diagram illustrating another application scenario of this application embodiment;

[0115] Figure 2A This is a schematic diagram of a monitoring system 200 according to an embodiment of this application;

[0116] Figure 2B This is a schematic diagram of another monitoring system 210 according to an embodiment of this application;

[0117] Figure 2C This is a schematic diagram of another monitoring system 220 according to an embodiment of this application;

[0118] Figure 3 This is a schematic diagram of a display process 300 according to an embodiment of this application;

[0119] Figure 4A This is a schematic diagram of an interactive interface according to an embodiment of this application;

[0120] Figure 4B This is a schematic diagram of another interactive interface according to an embodiment of this application;

[0121] Figure 5A This is a schematic diagram of another interactive interface according to an embodiment of this application;

[0122] Figure 5B This is a schematic diagram of another interactive interface according to an embodiment of this application;

[0123] Figure 5C This is a schematic diagram of another interactive interface according to an embodiment of this application;

[0124] Figure 5D This is a schematic diagram of another interactive interface according to an embodiment of this application;

[0125] Figure 6A This is a schematic diagram of another interactive interface according to an embodiment of this application;

[0126] Figure 6B This is a schematic diagram of another interactive interface according to an embodiment of this application;

[0127] Figure 6C This is a schematic diagram of another interactive interface according to an embodiment of this application;

[0128] Figure 6D This is a schematic diagram of another interactive interface according to an embodiment of this application;

[0129] Figure 6E This is a schematic diagram of another interactive interface according to an embodiment of this application;

[0130] Figure 6F This is a schematic diagram of another interactive interface according to an embodiment of this application;

[0131] Figure 7 This is a schematic diagram of a detection device 700 according to an embodiment of this application;

[0132] Figure 8AThis is a schematic diagram of a detection process 800 according to an embodiment of this application;

[0133] Figure 8B This is a schematic diagram illustrating a process for obtaining annotation information according to an embodiment of this application;

[0134] Figure 8C This is a schematic diagram illustrating a process for generating descriptive information according to an embodiment of this application;

[0135] Figure 8D This is a schematic diagram illustrating a process for generating a set of annotation information according to an embodiment of this application;

[0136] Figure 9A This is a schematic diagram of a video summary generation process according to an embodiment of this application;

[0137] Figure 9B This is a schematic diagram of another video summary generation process according to an embodiment of this application;

[0138] Figure 10 This is a schematic diagram of the structure of a device provided in an embodiment of this application. Detailed Implementation

[0139] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0140] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0141] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0142] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0143] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0144] In the embodiments of this application, the modules / components shown in the framework diagram (or structural diagram or system diagram) are merely examples of this application. The actual framework (or structure or system) may include more or fewer modules / components than those shown in the diagram, or may have different component configurations. Furthermore, the various components / modules shown in the diagrams may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0145] This application can be applied to any monitoring field, such as public safety monitoring (e.g., commercial area monitoring), traffic monitoring, environmental monitoring, health monitoring (e.g., home monitoring), industrial monitoring, etc., and this application does not limit it. The following provides an introduction to some monitoring fields.

[0146] Figure 1A This is a schematic diagram illustrating one application scenario of an embodiment of this application. Figure 1A The illustration shows a scenario for urban road monitoring (a scenario in the field of traffic monitoring).

[0147] Figure 1A In this system, monitoring poles can be installed on city roads, with cameras mounted on them (or the cameras can be mounted on fixed poles where traffic lights are located). Once activated, the cameras can record images of vehicles traveling on the city roads. Subsequently, based on the surveillance video data captured (or collected) by the cameras, monitoring can be conducted on issues such as speeding, illegal parking, uncivilized driving behaviors (such as using electronic devices or making phone calls while driving), whether trucks are carrying passengers, and whether they are carrying hazardous materials.

[0148] Figure 1B This is a schematic diagram illustrating another application scenario of this application embodiment. Figure 1B The image shows a shopping mall surveillance scenario (a scenario in the field of public safety surveillance).

[0149] Figure 1B Cameras can be installed on each floor of the shopping mall. Once activated, these cameras can capture images of the mall's environment. Subsequently, the captured (or collected) video data can be used to monitor aspects such as the operational status of escalators, dangerous behaviors on and around escalators, and unusual customers.

[0150] Figure 1CThis is a schematic diagram illustrating another application scenario of this application embodiment. Figure 1C The illustration shows a home monitoring scenario (a scenario in the field of health monitoring).

[0151] Figure 1C In this system, cameras can be installed in the elderly person's room and living room, as well as other spaces where they frequent. Once activated, these cameras can record footage of the elderly person in these areas. Subsequently, the video data captured by the cameras can be used to monitor whether the elderly person has fallen, fainted, or convulsed; whether they have engaged in dangerous behaviors such as touching hazardous materials; and whether they have taken their medication on time.

[0152] It should be understood that this application can also be applied to other application scenarios, such as fire monitoring, vegetation growth and health, and water level change scenarios in the field of environmental monitoring, and machine anomaly (such as stopping operation or abnormal shaking) monitoring scenarios in the field of industrial monitoring, etc., which will not be listed here.

[0153] It should be noted that the actions of the camera in collecting and transmitting surveillance video data, as well as the terminal equipment in storing, processing, and displaying surveillance video data, are all carried out in accordance with the relevant data protection laws and regulations of the country where the location is located, and with the authorization granted by the owner of the relevant device.

[0154] The following describes the monitoring systems applied in the aforementioned monitoring fields.

[0155] Figure 2A This is a schematic diagram of a monitoring system 200 according to an embodiment of this application. Figure 2A The monitoring system 200 in the middle only includes cameras.

[0156] Figure 2A The monitoring system 200 may include M1 (M1 is a positive integer) cameras: camera 1 (11), camera 2 (12), ..., camera M1 (1(M1)). Among them, camera 1 may include acquisition module 111, processing module 112 and display module 113; camera 2 may include acquisition module 121, processing module 122 and display module 123; ...; camera M1 may include acquisition module 1(M1)1, processing module 1(M1)2 and display module 1(M1)3.

[0157] Taking camera 1 as an example, the functions of each module are explained below. Acquisition module 111 can be used to acquire surveillance video data. Processing module 112 can be used to process the surveillance video data acquired by acquisition module 111, such as performing the detection method involved in this application. Display module 113 can be used for human-computer interaction, such as displaying a human-computer interaction interface (wherein, the processing results of processing module 112 can be displayed), receiving user operations, etc.

[0158] It should be understood that cameras 1 to M1 may also include other modules such as communication modules, and any two cameras among the M1 cameras can communicate with each other; this application does not limit this.

[0159] It should be understood that one camera can be used to collect one or more channels of surveillance video data, and M1 cameras can collect M1 or more channels of surveillance video data.

[0160] Figure 2B This is a schematic diagram of another monitoring system 210 according to an embodiment of this application. Figure 2B The monitoring system 210 includes cameras and edge computing devices.

[0161] Figure 2B The monitoring system 200 may include M2 ​​(M2 is a positive integer) cameras and N1 (N1 is a positive integer) edge computing devices. Among them, the M2 cameras include: camera 1 (21), camera 2 (22), ..., camera M2 (2 (M2)); the N1 edge computing devices include: edge computing device 1 (31), edge computing device 2 (32), ..., edge computing device N1 (3 (N1)).

[0162] For example, camera 1 may include acquisition module 211, processing module 212 and communication module 213; camera 2 may include acquisition module 221, processing module 222 and communication module 223; ...; camera M2 may include acquisition module 2(M2)1, processing module 2(M2)2 and communication module 2(M2)3.

[0163] Taking camera 1 as an example, the functions of each module are explained below. Acquisition module 211 can be used to acquire surveillance video data. Processing module 212 can be used to process the surveillance video data acquired by acquisition module 211. Communication module 213 can be used to transmit data, such as sending surveillance video data or the processing results of processing module 212 (such as processed surveillance video data or other information).

[0164] It should be understood that cameras 1 to M2 may also include other modules such as communication modules, and any two cameras among the M2 cameras can communicate with each other; this application does not limit this.

[0165] For example, edge computing device 1 may include a communication module 311, a processing module 312, and a display module 313; edge computing device 2 may include a communication module 321, a processing module 322, and a display module 323; ...; edge computing device N1 may include a communication module 3(N1)1, a processing module 3(N1)2, and a display module 3(N1)3. Any two edge computing devices among the N1 edge computing devices can communicate with each other.

[0166] Taking edge computing device 1 as an example, the functions of each module are explained below. Communication module 311 can be used to receive data, such as receiving monitoring video data sent by a camera or the processing results of processing module 212. Processing module 312 can be used to process the data sent by communication module 311. Display module 313 can be used for human-computer interaction, such as displaying a human-computer interaction interface (wherein, the processing results of processing module 312 can be displayed), receiving user operations, etc.

[0167] It should be understood that one camera can be used to collect one or more channels of surveillance video data, and M2 cameras can collect M2 or more channels of video data.

[0168] It should be understood that a camera can be connected to a single edge computing device, multiple cameras can be connected to a single edge computing device, or a camera can be connected to multiple edge computing devices; this application does not limit this.

[0169] In one possible implementation, the detection method involved in this application can be executed by the processing module of one of the N1 edge computing devices.

[0170] In one possible implementation, the detection method involved in this application (i.e., computing power sharing / computing power collaboration) can be executed collaboratively by the processing modules of multiple edge computing devices among N1 edge computing devices.

[0171] In one possible implementation, the detection method involved in this application can be executed collaboratively by the processing modules of one or more edge computing devices out of N1 edge computing devices and the processing modules of one or more cameras out of M2 cameras.

[0172] Figure 2C This is a schematic diagram of another monitoring system 220 according to an embodiment of this application. Figure 2C The monitoring system 220 includes cameras, edge computing devices, and servers.

[0173] Figure 2CThe monitoring system 200 may include M3 (M3 is a positive integer) cameras, N2 (N2 is a positive integer) edge computing devices, and H (H is a positive integer) servers. Among them, the M3 cameras include: camera 1 (41), camera 2 (42), ..., camera M3 (4 (M3)); the N2 edge computing devices include: edge computing device 1 (51), edge computing device 2 (52), ..., edge computing device N2 (5 (N2)); the H servers include: server 1 (61), server 2 (62), ..., server H (6H).

[0174] in, Figure 2C The modules and functions of the camera can be found in [reference]. Figure 2B The description will not be repeated here.

[0175] For example, edge computing device 1 may include a communication module 511 and a processing module 512; edge computing device 2 may include a communication module 521 and a processing module 522; ...; edge computing device N2 may include a communication module 5(N2)1 and a processing module 5(N2)2. Any two edge computing devices among the N2 edge computing devices can communicate with each other.

[0176] Taking edge computing device 1 as an example, the functions of each module are explained below. Communication module 511 can be used to receive data, such as receiving surveillance video data sent by a camera or the processing results of processing module 412. Processing module 512 can be used to process the data sent by communication module 511. Communication module 511 can also be used to send data, such as sending the processing results of processing module 412, surveillance video data, or the processing results of processing module 512.

[0177] For example, server 1 may include a communication module 611, a processing module 612, and a display module 613; server 2 may include a communication module 621, a processing module 622, and a display module 623; ...; server H may include a communication module 6H1, a processing module 6H2, and a display module 6H3. Any two of the H servers can communicate with each other.

[0178] Taking server 1 as an example, the functions of each module are explained below. The communication module 611 can be used to receive data, such as surveillance video data collected by a camera, the processing results of the processing module 412, or the processing results of the processing module 512. The processing module 612 can be used to process the data sent by the communication module 611. The display module can be used for human-computer interaction, such as displaying a human-computer interaction interface (wherein, the processing results of the processing module 612 can be displayed), receiving user operations, etc.

[0179] It should be understood that one camera can be used to collect one or more channels of surveillance video data, and M3 cameras can collect M3 or more channels of video data.

[0180] It should be understood that a camera can be connected to a single edge computing device, multiple cameras can be connected to a single edge computing device, or a camera can be connected to multiple edge computing devices; this application does not limit this.

[0181] It should be understood that there can be one edge computing device connected to one server, multiple edge computing devices connected to one server, or one edge computing device connected to multiple servers; this application does not limit this.

[0182] In one possible implementation, the detection method involved in this application can be executed by the processing module of one of the H servers.

[0183] In one possible implementation, the detection method involved in this application can be executed collaboratively by the processing modules of multiple servers out of H servers (i.e., computing power sharing / computing power collaboration).

[0184] In one possible implementation, the detection method involved in this application can be executed collaboratively by the processing modules of one or more of the H servers and the processing modules of one or more of the N2 edge computing devices.

[0185] In one possible implementation, the detection method involved in this application can be executed collaboratively by the processing modules of one or more servers among H servers, the processing modules of one or more edge computing devices among N2 edge computing devices, and the processing modules of one or more cameras among M3 cameras.

[0186] For example, the specific implementation of the server can be a cloud server, a physical (standalone) server, a server cluster, etc., and this application does not limit it.

[0187] The following is Figure 2B The monitoring system includes multiple cameras and an edge computing device. Taking the acquisition of one channel of monitoring video data by one camera as an example, the display method involved in this application will be explained.

[0188] Figure 3 This is a schematic diagram of a display process 300 according to an embodiment of this application. Figure 3 The display process 300 can be executed by an edge computing device.

[0189] S301, acquire surveillance video data, which includes multiple frames of images.

[0190] For example, an edge computing device can acquire surveillance video data collected by M2 cameras; that is, an edge computing device can acquire M2 channels of surveillance video data. Each channel of surveillance video data may include multiple frames of images.

[0191] It should be understood that edge computing devices can acquire surveillance video data from cameras in real time.

[0192] S302, Obtain the first detection command.

[0193] For example, the first detection instruction is used to instruct the edge computing device to detect surveillance video data to identify specific objects in the images included in the surveillance video data, and generate and display descriptive information of the specific objects. The specific objects may include specific people or specific items (such as specific animals, specific plants, specific vehicles, specific machines, etc.); for example, people wearing masks and hats, people holding controlled knives, illegally parked vehicles, speeding vehicles, people fighting, people illegally trespassing, lakes with water levels above the warning line, vegetation with a growth rate exceeding a threshold, machines operating at abnormal speeds, etc. This application illustrates the use of specific objects as abnormal objects (including objects with abnormal appearance and / or objects with abnormal behavior) as an example.

[0194] It should be noted that the characteristics of a specific object (which may include appearance and / or behavioral characteristics) can be determined based on the features of the object in a specific event. This allows for the subsequent detection of specific objects within the images included in the surveillance video data based on these characteristics. It should be understood that the definition of a specific event may differ across different surveillance domains or even within different application scenarios within the same surveillance domain.

[0195] For example, the first detection command can be triggered by either the user or the system. In one possible approach, the trigger time for the first detection command can be pre-set; thus, the system can trigger the first detection command each time the pre-set time arrives; the edge computing device can then receive the first detection command and execute S303. In another possible approach, the user can perform a detection operation on the edge computing device's interface to trigger the first detection command; the edge computing device can then receive the first detection command and execute S303.

[0196] For example, the first detection instruction can be used to instruct the edge computing device to detect one or more of the M2 channels of surveillance video data in order to detect a specific object in the image included in one or more of the M2 channels of surveillance video data (in other words, the surveillance video data targeted by the first detection instruction is one or more of the M2 channels of surveillance video data).

[0197] It should be understood that S301 and S302 can be viewed as two parallel steps.

[0198] S303, in response to the first detection command, displays descriptive information of the first target object in the image included in the monitoring video data.

[0199] For example, the first target object may refer to a specific object. Specifically, the first target object may refer to an object with a specific appearance and / or an object with specific behaviors.

[0200] For example, after receiving the first detection instruction, the edge detection device can respond to the first detection instruction by detecting the surveillance video data (the specific process will be explained later) to generate descriptive information of the first target object in the image included in the surveillance video data and display the descriptive information on the interactive interface.

[0201] From one perspective, the descriptive information of the first target object includes information describing the appearance of the first target object (also known as appearance description information) and / or information describing the behavior of the first target object (also known as behavior description information). It should be understood that when the first target object is an object with a specific appearance, the descriptive information of the first target object includes information describing the appearance of the first target object. When the first target object is an object with a specific behavior, the descriptive information of the first target object includes information describing the behavior of the first target object; alternatively, the descriptive information of the first target object includes information describing the behavior of the first target object and information describing the appearance of the first target object (in this case, the descriptive information of the first target object can also be referred to as the descriptive information of a specific event (hereinafter referred to as the first target event)). When the first target object is an object with both a specific appearance and specific behavior, the descriptive information of the first target object includes information describing the behavior of the first target object and information describing the appearance of the first target object (in this case, the descriptive information of the first target object can also be referred to as the descriptive information of the first target event).

[0202] From another perspective, the descriptive information of the first target object may include at least one of the following: identification information, attribute information, statistical information, feature information, thumbnail, or time information.

[0203] For example, the identification information may include an object identifier of the first target object and / or a behavior identifier of the first target object. The object identifier can be used to uniquely identify an object; for example, Person 1 identifies user 1, Person 2 identifies user 2, Vehicle 1 identifies vehicle 1, Vehicle 2 identifies vehicle 2, etc. The behavior identifier can be used to uniquely identify a behavior; for example, Behavior 1 identifies fighting, Behavior 2 identifies speeding, Behavior 3 identifies damaging a trash can, Behavior 4 identifies trespassing, Behavior 5 identifies loitering, etc.

[0204] For example, attribute information can be used to describe the attributes of a first target object, which can be a person or an item.

[0205] For example, the feature information may include information describing the appearance features of the first target object and / or information describing the behavioral features of the first target object. For instance, information describing the appearance features of the first target object may include "mask," "hat," "green dress," "white high heels," etc. As another example, information describing the behavioral features of the first target object may include "holding a stick," "damaging a trash can," "illegally entering," "loitering," etc.

[0206] For example, time information can refer to the timestamp of the image of the first target object appearing in the surveillance video data.

[0207] For example, a thumbnail may refer to a thumbnail of the image in the surveillance video data in which the first target object appears.

[0208] For example, the statistical information may include statistical information of the first target object (such as the frequency and duration of the first target object appearing in the surveillance video data), and statistical information of specific behaviors of the first target object (such as the frequency and duration of specific behaviors).

[0209] For example, the appearance description information may include at least one of the following: the object identifier of the first target object, the attributes of the first target object, the statistical information of the first target object, the appearance features of the first target object, the thumbnail of the first target object, or the timestamp of the image in which the first target object is located.

[0210] For example, the behavior description information may include at least one of the following: the object identifier of the first target object, the behavior identifier of the first target object, the attribute of the first target object, the statistical information of the specific behavior of the first target object, the behavior characteristics of the first target object, the thumbnail of the first target object, or the timestamp of the image where the first target object is located.

[0211] For example, the first target object can be one or more. The edge computing device can display multiple descriptive information on the interactive interface. One descriptive information can be the appearance description of a first target object, or the behavior description of a first target object, or the appearance description and behavior description of a first target object (i.e., the description of a first target event).

[0212] For example, a description (i.e., the appearance description of a first target object) is: [10:03:55~10:30:20][Person][Person 1 mask hat green dress white high heels].

[0213] For example, a description (i.e., the appearance description of a first target object) is: [00:15:25~00:30:20][Iitem][Vehicle 1 Green sports car license plate****].

[0214] For example, a descriptive message (i.e., a behavioral description of a first target object) is: [10:03:55~10:30:20][Person 3Behavior 1 Holding a stick].

[0215] For example, a descriptive message (i.e., a behavioral description of a first target object) is: [00:15:25~00:30:20][Person 10Behavior 2 speed 100km / h].

[0216] For example, a descriptive message (i.e., a behavioral description of a first target event) is: [12:22:01~13:45:30][Person][Person3Behavior 1 Mask, Hat, Black Jacket, Basketball Shoes, Holding a Stick].

[0217] Figure 4A This is a schematic diagram of an interactive interface according to an embodiment of this application. Figure 4A The detection result interface 400 may include one or more controls, including but not limited to: description information 401 of the first target object A1, description information 402 of the first target object A2, description information 403 of the first target object A3, description information 404 of the first target object A4, description information 405 of the first target object A5, ..., description information 406 of the first target object A20, description information 407 of the first target object A21, ..., and screen display box 408 (for displaying the monitoring video data), etc.

[0218] Figure 4B This is a schematic diagram of another interactive interface according to an embodiment of this application. Figure 4BThe detection result interface 410 may include one or more controls, including but not limited to: appearance description information 411 of the first target object A1, appearance description information 412 of the first target object A2, ..., behavior description information 413 of the first target object A8, behavior description information 414 of the first target object A10, ..., description information 415 of the first target event B1 (i.e., behavior description information + appearance description information of the first target object A20), ..., screen display box 416 (for displaying the monitoring video data), etc.

[0219] In summary, this application allows electronic devices (such as cameras, edge computing devices, servers, etc.) to detect surveillance video data, identifying objects (i.e., first target objects) with specific behaviors and / or appearances within the images, and displaying descriptive information about these first target objects. This eliminates the need for manual frame-by-frame review of the surveillance video data, resulting in high reliability and effectiveness. Furthermore, this application can detect one or more first target objects and / or one or more specific behaviors at a time, demonstrating high detection efficiency. In addition, displaying descriptive information about specific objects facilitates quick understanding of specific situations presented in the surveillance video data by users (such as monitoring / inspection personnel).

[0220] The following explanation uses the example of a user triggering the first detection command by performing a detection operation through an interactive interface.

[0221] Figure 5A This is a schematic diagram of another interactive interface according to an embodiment of this application.

[0222] Figure 5A In the monitoring interface 501, one or more controls may be included, including but not limited to: camera options 502, date setting options 503, screen display box 504, intelligent analysis options 505, filter information editing box 506, timeline 507, screen quantity setting options, search box, etc. This application does not limit these.

[0223] In one possible implementation, the user clicks the intelligent analysis option 505 (i.e., performs a detection operation) (at which point the edge computing device can obtain a first detection instruction). The edge computing device, in response to the user's action (i.e., in response to the obtained first detection instruction), displays descriptive information of the first target object in the image included in the surveillance video data. In this case, the surveillance video data targeted by the first detection instruction can be surveillance video data collected by all cameras, or surveillance video data collected by preset cameras. The time period corresponding to the surveillance video data targeted by the first detection instruction can be the current day, the previous day, the current week, the previous week, a preset time period, or other time periods; this application does not impose any restrictions on this.

[0224] In one possible implementation, the user can first select a target camera and then trigger a first detection command. In this case, the first detection command targets the surveillance video data collected by the target camera (hereinafter referred to as target video data). The time period corresponding to the surveillance video data targeted by the first detection command can be the current day, the previous day, the current week, the previous week, a preset time period, or other time periods; this application does not impose any restrictions on this. (See reference...) Figure 5B The user can sequentially click on the camera options corresponding to camera 1, camera 3, camera 4, camera 5, and camera 8 (at this time, the edge computing device can obtain the selection command). The edge computing device can respond sequentially to the user's operation (i.e., respond to the obtained selection command) by marking the camera options corresponding to camera 1, camera 3, camera 4, camera 5, and camera 8 in the monitoring interface 504, and sequentially obtaining the camera identifiers of camera 1, camera 3, camera 4, camera 5, and camera 8. Next, the user can click on the intelligent analysis option 505. The edge computing device can then respond to the user's operation by displaying descriptive information of the first target object in the image included in the target monitoring video data collected by the target camera (determined based on the obtained camera identifier).

[0225] In one possible approach, the user can first set a detection time period and then trigger the first detection command. In this case, the surveillance video data targeted by the first detection command can be surveillance video data collected by all cameras, or surveillance video data collected by pre-set preset cameras. The time period corresponding to the surveillance video data targeted by the first detection command is the detection time period. (See reference...) Figure 5CThe user can select "March 14, 2024" in the date setting option 503 (at this time, the edge device can receive the time setting instruction). The edge computing device, responding to the user's action (i.e., responding to the time setting instruction), displays the year and month ("March 2024") and highlights the day ("14") in the date setting option 503, as well as obtaining the start and end dates of the detection (both start and end dates are March 14, 2024). Next, the user can drag the progress bar on the timeline 507 to "12" (at this time, the edge device can receive the time setting instruction). The edge computing device, responding to the user's action (i.e., responding to the time setting instruction), moves the progress bar to "12" and stops, obtaining the start and end times of the detection (i.e., the start time is 12:00:00, and the stop time is 24:00:00). The start and end dates and times of the detection can form a detection time period. Subsequently, the user can click on the intelligent analysis option 505, and the edge computing device can respond to the user's operation by displaying descriptive information of the first target object in the images included in the monitored video data within the detection period. It should be understood that the user can also select two dates in the date setting option 503, as the start and end dates of the detection, respectively.

[0226] It should be understood that the date setting option 503 and the duration setting option (timeline is one example) in the monitoring interface 501 can be in any form; thus, users can select the start and end dates and start and end times of the detection in any way; in addition, users can also directly input the detection time period in text form.

[0227] In one possible approach, the user can first select the target camera and set the detection time period, and then trigger the first detection command. In this case, the first detection command targets the target surveillance video data, and the time period corresponding to the surveillance video data targeted by the first detection command is the detection time period. (See reference...) Figure 5DUsers can sequentially click on the camera options corresponding to camera 1, camera 2, camera 6, and camera 7. The edge computing device can respond to the user's actions by marking the camera options corresponding to camera 1, camera 2, camera 6, and camera 7 in the monitoring interface 501, and sequentially obtaining the camera identifiers of camera 1, camera 2, camera 6, and camera 7. Next, users can select the date option "March 12, 2024" in the date setting option 503. The edge computing device can respond to the user's actions by displaying the year and month ("March 2024") and highlighting the day ("12") in the date setting option 503, and obtaining the start and end dates of the detection (both start and end dates are March 12, 2024). Next, the user can drag the progress bar on timeline 507 to "6". The edge computing device can respond to the user's operation by stopping the progress bar at "6" and obtaining the start and end times of the detection (i.e., start time is 6:00:00, stop time is 24:00:00). Subsequently, the user can click the intelligent analysis option 505, and the edge computing device can respond to the user's operation by displaying descriptive information of the first target object in the images included in the target surveillance video data within the detection period. It should be understood that this application does not restrict whether the user selects the target camera first or sets the detection period first.

[0228] In one possible approach, the user can first enter filter information and then trigger the first detection command. (See reference...) Figure 5D Users can input filtering information (which may include attribute information, feature information, thumbnails, and statistical information) in the filtering information editing box 506. The edge computing device can respond to the user's operation and obtain the filtering information. Subsequently, the user can click the intelligent analysis option 505, and the edge computing device can respond to the user's operation and display the description information of the first target object in the image included in the monitoring video data. In this case, the first target object is obtained based on the filtering information; the monitoring video data targeted by the first detection instruction can be monitoring video data collected by all cameras, or monitoring video data collected by preset cameras. In addition, the time period corresponding to the monitoring video data targeted by the first detection instruction can be the current day, the previous day, the current week, the previous week, a preset time period, or other time periods, which are not limited in this application.

[0229] It should be understood that users can first perform any two or three actions from inputting filtering information, selecting the target camera, and setting the detection time period, and then trigger the first detection command; for details, please refer to the description above, which will not be repeated here.

[0230] For example, in response to the first detection command received, the edge computing device can display not only the description information of the first target object, but also the playback options corresponding to the target video segment, which consists of multiple frames of images of the first target object contained in the monitoring video data.

[0231] Figure 6A This is a schematic diagram of another interactive interface according to an embodiment of this application.

[0232] Reference Figure 6A The detection results interface 601 may include one or more controls, including but not limited to: playback options 602, 603, and 604, a screen display box (for displaying the monitoring video data), a filter information editing box 611, and intelligent analysis options 612. The playback options include descriptive information and a play button.

[0233] Continue to refer to Figure 6A There are multiple playback options 602, each corresponding to a target video segment for the same first target object. A target video segment for a first target object consists of images containing that first target object within the monitoring video data. Each playback option 602 includes a description 605 (i.e., a description of the appearance of a first target object) and a play button 606. The description 605 may sequentially include: time information, attribute information, object identifier, and appearance features. The description 605 serves as a summary of the playback option 602, allowing users (such as monitoring / inspection personnel) to obtain the main information about the appearance of the first target object in the target video segment. When the user clicks the play button 606 (the edge computing device can receive the playback command), the edge computing device responds to the user's action (i.e., responds to the received playback command) and plays the target video segment corresponding to playback option 602 in the display frame.

[0234] Continue to refer to Figure 6AThere are multiple playback options 603, each corresponding to a target video segment for the same first target object. A target video segment consists of images of a first target object contained within the monitoring video data. Each playback option 603 includes a description 607 (i.e., a behavioral description of the first target object) and a play button 608. The description 607 may include, in sequence, time information, attribute information, behavioral identifier, and behavioral characteristics. The description 607 serves as a summary of the playback option 603, allowing users (such as monitoring / inspection personnel) to obtain the main information about the first target object's behavior in the target video segment. When the user clicks the play button 608 (the edge computing device can receive the playback command), the edge computing device responds to the user's action (i.e., responds to the received playback command) and plays the target video segment corresponding to playback option 603 in the display frame.

[0235] Continue to refer to Figure 6A There are multiple playback options 604, each corresponding to a target video segment for the same first target object. A target video segment consists of images of a first target object contained within the monitoring video data. Each playback option 604 includes a description 609 (i.e., a description of the behavior of a first target object and a description of the behavior of a first target event) and a play button 610. The description 609 may sequentially include: thumbnail, time information, attribute information, object identifier, behavior identifier, appearance features, and behavior features. The description 609 serves as a summary of the playback option 604, allowing users (such as monitoring / inspection personnel) to obtain key information about the appearance and behavior of the first target object in the target video segment. When the user clicks the play button 610 (the edge computing device can receive the playback command), the edge computing device responds to the user's action (i.e., responds to the received playback command) and plays the target video segment corresponding to playback option 604 in the display frame.

[0236] Figure 6B This is a schematic diagram of another interactive interface according to an embodiment of this application. Wherein, Figure 6B The interactive interface is in Figure 6A Based on the above, relative to Figure 6A Regarding the detection results interface 601, Figure 6B The detection result interface 601 also includes camera identifiers. Each playback option may also include a camera identifier, meaning that the target video segment corresponding to each playback option is composed of images captured by the camera corresponding to that camera identifier (or, in other words, the user (such as monitoring / inspection personnel) can identify the camera used to capture the target monitoring video segment corresponding to that playback option based on the camera identifier).

[0237] For example, there may be situations where multiple cameras capture the same object; therefore, the same first target object can correspond to target video segments captured by multiple cameras. Figure 6B Multiple playback options include the same descriptive information, meaning that multiple playback options include the appearance and / or behavior descriptions of the same first target object. For example, Figure 6B The description information included in the first three playback options 602 is the appearance description information of the first target object identified by Person 1. For example, Figure 6B The first two playback options 603 include descriptions of the behavior of the first target object identified by Person 3 (descriptions of the behavior identified by Behavior 1). For example, Figure 6B The first two playback options 604 include descriptions of the appearance of the first target object identified by Person 3 and descriptions of its behavior (descriptions of the behavior identified by Behavior 1).

[0238] Figure 6C This is a schematic diagram of another interactive interface according to an embodiment of this application. Wherein, Figure 6C The interactive interface is in Figure 6A Based on the above, relative to Figure 6A Regarding the detection results interface 601, Figure 6C The detection results interface 601 also includes camera identifiers. Each playback option may also include one or more camera identifiers.

[0239] When a playback option includes a camera identifier, it means that the target video segment corresponding to that playback option was captured by the camera corresponding to that camera identifier. For example, Figure 6C The second playback option 602 includes a camera identifier, namely camera 1. In other words, the target monitoring segment corresponding to the second playback option 602 is captured by the camera identified by camera 1.

[0240] When a playback option includes multiple camera identifiers, it means that the target video segment corresponding to that playback option was captured by the cameras corresponding to those multiple camera identifiers. For example, Figure 6C The first playback option 602 includes three camera identifiers: Camera 1, Camera 3, and Camera 5; that is, the target surveillance footage corresponding to the first playback option 602 was captured by the cameras identified by Camera 1, Camera 3, and Camera 5. For example, Figure 6CThe first playback option 603 includes two camera identifiers: Camera 13 and Camera 18; that is, the target surveillance footage corresponding to the first playback option 603 was captured by the camera identified by Camera 13 and the camera identified by Camera 18. For example, Figure 6C The first playback option 604 includes two camera identifiers: camera 13 and camera 18; that is, the target monitoring segment corresponding to the first playback option 604 is captured by the camera identified by camera 13 and the camera identified by camera 18.

[0241] In one possible implementation, when the playback option includes multiple camera identifiers, the target video segment corresponding to the playback option is composed of target video segments captured by the cameras corresponding to these multiple camera identifiers in a temporal sequence; the images included in the target video segment corresponding to the playback option are single-view images. In this case, when the user clicks the play button 606, the edge computing device can respond to the user's operation and play the single-view image of the target video segment in the screen display box.

[0242] In one possible implementation, when the playback option includes multiple camera identifiers, the target video segment corresponding to the playback option is stitched together from the target video segments captured by the cameras corresponding to these multiple camera identifiers according to their viewpoints (this can be achieved using augmented reality / virtual reality technology); the monitoring video segment corresponding to the playback option includes panoramic images (or multi-view images). In this case, when the user clicks the play button 606, the edge computing device can respond to the user's operation and play the panoramic image (or multi-view image) of the target video segment in the display frame.

[0243] For example, in response to the first detection command received, the edge computing device can display a video summary in addition to displaying descriptive information of the first target object.

[0244] Figure 6D This is a schematic diagram of another interactive interface according to an embodiment of this application. Wherein, Figure 6D The interactive interface is in Figure 6A Based on the above, relative to Figure 6A Regarding the detection results interface 601, Figure 6D The detection results interface 601 also includes a video summary (also known as a video summary).

[0245] For example, an edge computing device can generate a video summary based on the description information of all first target objects. In one possible implementation, the video summary may include all or part of the description information of all first target objects. In another possible implementation, the video summary may include all or part of the description information of a portion of the first target objects; wherein, this portion of the first target objects may be first target objects whose behavior meets a first preset condition (which can be set as needed), or it may be first target objects whose corresponding target video segment duration exceeds a preset duration; this application does not impose any limitations on this. In one possible implementation, the video summary may be a summary of the overall surveillance video data. Optionally, the video summary may also include processing suggestions for the first target objects.

[0246] In this way, users (such as monitoring / inspection personnel) can learn about specific situations in all the monitoring video data being inspected, as well as processing suggestions, based on the video summary.

[0247] It should be understood that the video summary of this application may include multiple statements or only one statement, and this application does not limit this.

[0248] It should be understood that in another possible scenario, Figures 6A to 6D In the detection results interface, one playback option corresponds to the target video segments (not shown in the figure) of multiple first target objects with the same behavior.

[0249] Figure 6E This is a schematic diagram of another interactive interface according to an embodiment of this application. Wherein, Figure 6E The interactive interface is in Figure 6D Based on the above, relative to Figure 6D Regarding the detection results interface 601, Figure 6E The description information for playback option 604 in the detection results interface 601 also includes statistical information. For example... Figure 6E In playback option 604, "Person 3 appeared a total of 3 times" is the statistical information of the first target object, and "the total duration of the fight was 20 minutes" is the statistical information of the specific behavior of the first target object.

[0250] In this way, users (such as monitoring / inspection personnel) can quickly grasp the general situation of the target video segment corresponding to each playback option based on statistical information.

[0251] Figure 6F This is a schematic diagram of another interactive interface according to an embodiment of this application.

[0252] Figure 6FIn the detection results interface 601, one or more controls may be included, including but not limited to: playback options 613, a screen display box (for displaying the monitoring video data), a filter information editing box 611, and intelligent analysis options 612. The playback options include description information, camera location, and a play button.

[0253] Continue to refer to Figure 6F There are multiple playback options 613, each corresponding to a specific target video segment for a given first target object (where each first target object can have one or more target video segments, and a single target video segment consists of multiple consecutive frames). These multiple playback options 613 can be displayed in chronological order. This integrates all the target video segments corresponding to the first target object into a timeline for easy viewing by the user.

[0254] It should be understood that, Figures 6A to 6F The format of the descriptive information and video summary in the detection results interface is only an example. The descriptive information and video summary in the detection results interface of this application can also be displayed in other formats, and this application does not limit them.

[0255] For example, the format of a video summary is: [Time Information] + [Appearance Features] + [Behavioral Features] + [Object Identifier].

[0256]

[0257] For example, during the detection of each frame of surveillance video data, when the edge computing device detects a first target object in a certain frame, it can display that frame in the display window of the monitoring interface 501 and highlight the first target object in that frame (e.g., highlight the outline of the first target object, zoom in on the first target object, etc.) to draw the user's (e.g., monitoring / inspection personnel's) attention. Optionally, it can also display some or all of the descriptive information of the first target object in that frame in the display window of the monitoring interface 501, so that the user (e.g., monitoring / inspection personnel) can promptly obtain the main information of the first target object.

[0258] For example, the monitoring interface 501 can also display the detection progress of the edge computing device on the monitoring video data.

[0259] It should be noted that the detection result interfaces in different monitoring fields can be different (e.g., different backgrounds, different options, different layouts, different font styles, etc.) or the same.

[0260] It should be noted that the monitoring interface can be different for users with different permissions (such as monitoring / inspection personnel) within the same monitoring domain.

[0261] Reference Figures 6A to 6F In any of the diagrams shown, after the edge computing device responds to the first detection command and displays the description information of the first target object, the user can enter filter information in the filter information editing box 611. The edge computing device can then obtain the filter information in response to the user's operation. Subsequently, the user can click the intelligent analysis option 612 (at this time, the edge computing device can receive the second detection command). The edge computing device can then respond to the user's operation (i.e., in response to the obtained second detection command) and display the description information of the second target object in the image included in the target video segment. In other words, the edge computing device, in response to the obtained second detection command, detects the target video segment based on the filter information, detects the description information of the second target object in the image included in the target video segment, and displays it. It can be seen that the second target object is obtained by filtering from the first target object based on the filter information; that is, the second target object is a subset of the first target object; the description information of the second target object can be a subset of the description information of the first target object.

[0262] It should be understood that the edge computing device can refresh the detection result interface 601, displaying descriptive information of the second target object in the image included in the target video segment, and... Figures 6A to 6F Similarly, I will not go into details here.

[0263] The following describes the process by which edge computing devices detect surveillance video data.

[0264] Figure 7 This is a schematic diagram of a detection device 700 according to an embodiment of this application.

[0265] Figure 7 The detection device 700 may include a data acquisition module 701, an identification module 702, and an integration module 703. The detection device 700 can be used to perform the detection method involved in this application.

[0266] Reference Figure 7 and Figure 2A The identification module 702 and integration module 703 in the detection device 700 can be deployed in the processing module of each camera, and the data acquisition module 701 can be the acquisition module of each camera.

[0267] Reference Figure 7 and Figure 2BThe detection device 700 can be deployed in the processing module of one edge computing device. Alternatively, the detection device 700 can be deployed in the processing modules of multiple edge computing devices; in this case, each edge computing device may include at least one of the data acquisition module 701, recognition module 702, and integration module 703 of the detection device 700. Alternatively, the detection device 700 can be deployed in the processing modules of one or more edge computing devices and the processing modules of one or more cameras; in this case, each edge computing device may include at least one of the data acquisition module 701, recognition module 702, and integration module 703 of the detection device 700, and each camera may include at least one of the data acquisition module 701, recognition module 702, and integration module 703 of the detection device 700.

[0268] Reference Figure 7 and Figure 2C The detection device 700 can be deployed in the processing module of a server. Alternatively, the detection device 700 can be deployed in the processing modules of multiple servers; in this case, each server may include at least one module of the detection device 700, namely the data acquisition module 701, the identification module 702, and the integration module 703. Alternatively, the detection device 700 can be deployed in the processing modules of one or more servers, and in the processing modules of one or more edge computing devices; in this case, each edge computing device and each server may include at least one module of the detection device 700, namely the data acquisition module 701, the identification module 702, and the integration module 703. Alternatively, the detection device 703 can be deployed in the processing modules of one or more servers, and in the processing modules of one or more edge computing devices, and in the processing modules of one or more cameras; in this case, each edge computing device and each server may include at least one module of the detection device 700, namely the data acquisition module 701, the identification module 702, and the integration module 703, and each camera may include at least one module of the detection device 700, namely the data acquisition module 701, the identification module 702, and the integration module 703.

[0269] The following is Figure 2B Taking the monitoring system as an example, which includes multiple cameras and an edge computing device, and the detection device 700 is deployed in an edge computing device, the detection method performed by the detection device 700 is explained.

[0270] Figure 8A This is a schematic diagram of a detection process 800 according to an embodiment of this application.

[0271] S801, acquire surveillance video data.

[0272] For example, the data acquisition module 701 can acquire the surveillance video data collected by the camera.

[0273] For example, when the user only clicks the intelligent analysis option 505 in the monitoring interface 501, the monitoring video data acquired by the data acquisition module 701 can be monitoring video data collected by all cameras, or monitoring video data collected by preset cameras. Furthermore, the time period corresponding to the monitoring video data acquired by the data acquisition module 701 can be the current day, the previous day, the current week, the previous week, a preset time period, or other time periods; this application does not impose any restrictions on this.

[0274] For example, when a user clicks the camera option and the intelligent analysis option 505 in the monitoring interface 501, the monitoring video data acquired by the data acquisition module 701 can be the monitoring video data collected by the camera corresponding to the camera option clicked by the user (i.e., the target monitoring video data). Furthermore, the time period corresponding to the target monitoring video data acquired by the acquisition module 701 can be the current day, the previous day, the current week, the previous week, a preset time period, or other time periods; this application does not impose any restrictions on this.

[0275] For example, when the user sets the date (and / or drags the timeline) in the date setting option 503 of the monitoring interface 501 and clicks the intelligent analysis option 505, the monitoring video data acquired by the data acquisition module 701 can be the monitoring video data within the detection time period (the monitoring video data can be the monitoring video data collected by all cameras, or the monitoring video data collected by the preset cameras).

[0276] For example, when a user clicks the camera option in the monitoring interface 501, sets the date in the date setting option 503 (and / or drags the time axis) and the intelligent analysis option 505, the monitoring video data acquired by the data acquisition module 701 can be the target monitoring video data within the detection time period.

[0277] For example, after the data acquisition module 701 acquires the surveillance video data, on the one hand, the data acquisition module 701 can output the surveillance video to the display module, which then displays some or all of the images included in the surveillance video data collected by the camera in the screen display frame of the monitoring interface. On the other hand, the data acquisition module 701 can output the surveillance video data to the recognition module 702, which then executes S802.

[0278] S802, obtain the annotation information of the first target object in the image included in the surveillance video data.

[0279] For example, AI algorithms can be used to generate annotation information for the first target object in the images included in the surveillance video data.

[0280] For example, the following describes the process of obtaining the annotation information of the first target object in the i-th frame of the surveillance video data, using the i-th frame as an example. Refer to S8021 to S8025 below:

[0281] S8021, Identify the object in the i-th frame of the image, where the initial value of i is 1.

[0282] For example, the recognition module 702 can invoke deep learning object detection algorithms (such as YOLOv8 (You Only Look Once version 8), Faster R-CNN (Faster Region-based Convolutional Neural Network), MobileNet (Lightweight Network), Tiny YOLO (Tiny You Only Look Once), etc.) to recognize objects in the images included in the surveillance video data, thereby identifying the objects in the images included in the surveillance video data. Specifically, in step S8021, the coordinates of the bounding box of the object in the i-th frame image and the object's attributes can be obtained.

[0283] Where i is a positive integer, the initial value of i is 1, and the maximum value of i is the total number of frames K (K is a positive integer) of the images included in the surveillance video data.

[0284] S8022, determine whether an object with a specific appearance is identified from the objects in the i-th frame image.

[0285] For example, the recognition module 702 can call an image recognition model and combine it with a predefined appearance feature library (wherein the appearance feature library may include multiple sets of appearance features, each set of appearance features can be determined according to the appearance features of the object in a specific event) to recognize the appearance of the object in the i-th frame image, so as to determine whether there is an object with a specific appearance in the object in the i-th frame image.

[0286] It should be noted that the image recognition model involved in this application can be obtained by optimizing (or fine-tuning) an existing trained image recognition model using the training data of this application. The training data of this application may include training images and reference annotation information of the first target object in the training images; the training images can be input into the image recognition model, which performs forward computation to output annotation information; subsequently, the annotation information output by the image recognition model can be compared with the reference annotation information to adjust the model parameters of the image recognition model.

[0287] If an object with a specific appearance is identified from the objects in the i-th frame image, then S8023 is executed; if no object with a specific appearance is identified from the objects in the i-th frame image, then S8024 is executed.

[0288] S8023, if an object with a specific appearance is identified from the objects in the i-th frame image, then the annotation information of the object with the specific appearance is generated.

[0289] For example, the annotation information (which may also be called appearance annotation information) of an object with a specific appearance (i.e., the first target object) may include at least one of the following: the object identifier of the first target object, the attributes of the first target object, time information (i.e., the timestamp of the i-th frame image), the appearance features of the first target object, and the thumbnail of the i-th frame image.

[0290] For example, the i-th frame image, along with the coordinates of the bounding box of the object in the i-th frame image and the object's attributes, can be input into the image recognition model. The image recognition model can then combine a predefined appearance feature library with the coordinates of the bounding box of the object in the i-th frame image and the object's attributes to recognize the appearance of the object in the i-th frame image. When the image recognition model recognizes an object with a specific appearance, it can output the annotation information of the object with that specific appearance; when the image recognition model does not recognize an object with that specific appearance, it can output other information.

[0291] S8024, determine whether an object with a specific behavior is identified from the objects in the i-th frame image.

[0292] For example, the recognition module 702 can call a behavior recognition model (such as 3D CNN (3D Convolutional Neural Network) or LSTM (Long Short-Term Memory)) and combine it with a predefined behavior feature library (wherein the behavior feature library may include multiple sets of behavior features, each set of behavior features can be determined according to the behavior features of objects in a specific event) to recognize the behavior of objects in the i-th frame image, so as to determine whether there are objects with specific behaviors in the i-th frame image.

[0293] If an object with a specific behavior is identified from the objects in the i-th frame image, then S8025 is executed; if no object with a specific behavior is identified from the objects in the i-th frame image, then S8026 is executed.

[0294] S8025, if an object with a specific behavior is identified from the objects in the i-th frame image, then the annotation information of the object with the specific behavior is generated.

[0295] For example, the i-th frame image, the coordinates of the bounding box of the object in the i-th frame image, the object's attributes, and the appearance features of the first target object can be input into the behavior recognition model. The behavior recognition model combines a predefined behavior feature library with the coordinates of the bounding box of the object in the i-th frame image, the object's attributes, and the appearance features of the first target object to recognize the behavior of the object in the i-th frame image. When the behavior recognition model recognizes an object with a specific behavior, it can output the annotation information of the object with that specific behavior; when the image recognition model does not recognize an object with a specific behavior, it can output other information.

[0296] For example, the annotation information (which may also be called behavior annotation information) of an object with a specific behavior (i.e., the first target object) may include at least one of the following: the object identifier of the first target object, the behavior identifier of the first target object, the attributes of the first target object, time information (i.e., the timestamp of the i-th frame image), the behavior features of the first target object, and the thumbnail of the i-th frame image.

[0297] The first target object includes objects with a specific appearance and / or objects with specific behaviors. The annotation information of objects with a specific appearance and / or objects with specific behaviors can constitute the annotation information of the first target object.

[0298] S8026, determine whether i is less than or equal to K.

[0299] For example, when i is less than or equal to K, S8027 is executed; when i is greater than K, the identification module 702 can execute the process of S802 and output the annotation information of the first target object to the integration module 703, which then executes S803.

[0300] S8027, increment i by 1.

[0301] For example, after S8027 is executed, the execution can return to S8021.

[0302] S803, Based on the annotation information of the first target object, generate the description information of the first target object.

[0303] For example, AI algorithms can be used to generate descriptive information about the first target object.

[0304] For example, S803 may include the following steps S8031 to S8033:

[0305] S8031, concatenate the annotation information of the same first target object to obtain the annotation information set of each first target object.

[0306] For example, after the recognition module 702 recognizes a first target object from the image, the generated annotation information of the first target object can be called a annotation information; in this way, the recognition module 702 can output G (G is a positive integer) annotation information to the integration module 703.

[0307] S8031 may include the following steps S1 to S5:

[0308] S1, determine whether the object identifier in the j-th annotation information exists in the database, where the initial value of j is 1.

[0309] For example, if the object identifier in the j-th annotation information exists in the database, then S2 is executed; if the object identifier in the j-th annotation information does not exist in the database, then S3 is executed.

[0310] S2, if the object identifier in the j-th annotation information exists in the database, then add the j-th annotation information to the annotation information set corresponding to the object identifier in the j-th annotation information.

[0311] S3. If there is no object identifier in the database that matches the j-th annotation information, then create the object identifier and the corresponding annotation information set in the database for the j-th annotation information, and add the j-th annotation information to the annotation information set corresponding to the object identifier in the j-th annotation information.

[0312] S4, determine whether j is less than or equal to G.

[0313] For example, if j is less than or equal to G, then S5 is executed; if j is greater than G, then the process of S8031 ends, and then the integration module 703 executes S8032.

[0314] S5, increment j by 1.

[0315] For example, after executing S5, you can return to execute S1.

[0316] For example, the integration module 703 can call deep learning algorithms (such as SORT (Simple Online and Realtime Tracking) and DeepSORT (Deep Simple Online and Realtime Tracking)) to implement the above S1 to S5.

[0317] It should be understood that the annotation information set corresponding to any object identifier may include only the appearance annotation information of the first target object corresponding to the object identifier, or only the behavior annotation information of the first target object corresponding to the object identifier, or both the appearance annotation information and the behavior annotation information of the first target object corresponding to the object identifier.

[0318] In this way, multiple associations (temporal association and spatial association) can be made with the same primary target object to generate an event chain of the same primary target object (i.e., the annotation information of the same primary target object).

[0319] S8032, based on the annotation information set of each first target object, generate statistical information of each first target object and / or statistical information of specific behaviors of each first target object.

[0320] For example, the integration module 703 can perform information statistics based on the annotation information set of each first target object to generate statistical information of each first target object and / or statistical information of specific behaviors of each first target object.

[0321] S8033, integrate the annotation information set and statistical information of each first target object to obtain the descriptive information of each first target object.

[0322] For example, for each first target object, the integration module 703 can perform deduplication, simplification, and other processing on the annotation information in the annotation information set of the first target object to obtain the integrated annotation information of the first target object; then, the integration module 703 can use the integrated annotation information and statistical information of the first target object (including the statistical information of each first target object and / or the statistical information of the specific behavior of each first target object) to generate the description information of the first target object.

[0323] In this way, the automatic identification of the first target object and the generation of its description by electronic devices (such as cameras, edge computing devices, servers, etc.) can significantly reduce the time spent manually searching frame by frame, improve the efficiency of video information filtering, and reduce the risk of information omission.

[0324] For example, S802 to S803 can be executed using an end-to-end AI algorithm, that is, the monitoring video data is input into the end-to-end AI algorithm, and the AI ​​algorithm outputs the description information of the first target object.

[0325] Figure 9A This is a schematic diagram of a video summary generation process according to an embodiment of this application.

[0326] In one possible approach, the integration module 703 of the edge computing device can also generate a video summary based on the annotation information of the first target object.

[0327] For example, the integration module 703 can invoke an NLP (Natural Language Processing) model (such as BERT (Bidirectional Encoder Representations from Transformers), T5 (Text-To-Text Transfer Transformer), or GPT (Generative Pre-trained Transformer)) to generate a video summary. For example, after receiving the first detection instruction, the integration module 703 can input the annotation information of the first target object output by the recognition module 702 (and / or the statistical information of the first target object generated by the integration module 703 and / or the statistical information of the specific behavior of the first target object) into the NLP model, which will then process the annotation information of the first target object to obtain the video summary.

[0328] Figure 9B This is a schematic diagram of another video summary generation process according to an embodiment of this application.

[0329] In one possible approach, the integration module 703 of the edge computing device can also generate a video summary based on the description information of the first target object.

[0330] For example, the integration module 703 can call an NLP model (such as BERT, T5, GPT) to generate a video summary. For example, after receiving the first detection instruction, the integration module 703 can input the description information of the first target object generated by the integration module 703 into the NLP model, and the NLP model will process the description information of the first target object to obtain a video summary.

[0331] It should be understood that embodiments of this application can also employ structured templates to automatically generate video summaries. This eliminates the need to call NLP models, reducing the computational power consumption of edge computing devices. The structured module can be configured by the system or user-defined; and this application also supports custom extensions of the structured template.

[0332] For example, the integration module 703 of the edge computing device can also extract the corresponding image from the monitoring video data based on the time information in the description information of the first target object, and generate the target video segment.

[0333] For example, the video summary can be a summary of the overall surveillance video data or a summary of all the first target objects.

[0334] In one possible approach, the integration module 703 of the edge computing device can also generate a video summary based on the description information of the second target object. The specific generation method can refer to the method described above for generating a video summary based on the description information of the first target object, and will not be repeated here.

[0335] Furthermore, edge computing devices can acquire other sensor data associated with surveillance video data. Then, the edge computing devices can combine this other sensor data to detect the surveillance video data and generate descriptive information about the first target object. This other sensor data may include, but is not limited to, audio data, radar data, infrared data, data collected by temperature and vibration sensors, and data collected by smoke sensors. In this way, combining data from other modalities to determine the first target object in the images included in the surveillance video data, as well as the descriptive information of the first target object, can improve the accuracy of the identified first target object and the accuracy and comprehensiveness of the generated descriptive information.

[0336] Similarly, this application can also combine data from other modalities to determine the second target object and its descriptive information, thereby improving the accuracy of the identified second target object and the accuracy and comprehensiveness of the generated descriptive information of the second target object.

[0337] In one example, Figure 10 The schematic block diagram illustrating an embodiment of the present application shows an apparatus 1000. The apparatus 1000 may include a processor 1001 and a transceiver 1002, and optionally, a memory 1003.

[0338] The various components of device 1000 are coupled together via bus 1004, which includes a data bus, a power bus, a control bus, and a status signal bus. However, for clarity, all buses are referred to as bus 1004 in the figure.

[0339] Optionally, the memory 1003 can be used to store instructions from the foregoing method embodiments. The processor 1001 can be used to execute the instructions in the memory 1003, control the transceiver 1002 to receive signals, and control the transceiver 1002 to transmit signals.

[0340] The device 1000 may be an electronic device or a chip of an electronic device as described in the above method embodiments. The electronic device may be a camera, an edge computing device, or a server.

[0341] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0342] This application also provides a chip including one or more interface circuits and one or more processors; the one or more processors receive or transmit data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the above-described related method steps that implement the method in the above embodiments are executed. The interface circuit is a transceiver 1002.

[0343] This application also provides a computer-readable storage medium storing computer instructions. When these computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the methods described in the above embodiments. Exemplarily, the computer-readable storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0344] This application also provides a computer program product comprising computer instructions that, when executed by a computer or processor, cause the computer to perform the aforementioned steps to implement the methods described in the above embodiments. Exemplarily, the computer program product may be stored in random access memory (RAM), flash memory, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, read-only optical discs (CD-ROMs), or any other form of storage medium known in the art.

[0345] In this embodiment, the electronic device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0346] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0347] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0348] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0349] Any content in the various embodiments of this application, as well as any content in the same embodiment, can be freely combined. Any combination of the above content is within the scope of this application.

[0350] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A display method, characterized in that, Applied to the field of monitoring, the method includes: Acquire surveillance video data, which includes multiple frames of images; Obtain the first detection command; In response to the first detection command, the description information of a first target object in the image included in the surveillance video data is displayed, wherein the first target object is one or more.

2. The method according to claim 1, characterized in that, The description information of the first target object includes information describing the appearance of the first target object and / or information describing the behavior of the first target object.

3. The method according to claim 1, characterized in that, The description information of the first target object includes at least one of the following: identification information, attribute information, statistical information, feature information, thumbnail, or time information.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: In response to the first detection command, a video summary of the surveillance video data is displayed.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: In response to the first detection command, the playback options corresponding to the target video segment are displayed, wherein the target video segment consists of multiple frames of images containing the first target object in the monitoring video data.

6. The method according to claim 5, characterized in that, There are multiple playback options, which are displayed in chronological order.

7. The method according to claim 5, characterized in that, Each target video segment corresponding to the same primary target object has one playback option.

8. The method according to claim 5, characterized in that, Multiple first target objects with the same behavior correspond to a single playback option for their respective target video segments.

9. The method according to any one of claims 5 to 8, characterized in that, The playback options are multiple, and the method further includes: In response to a playback command, a target playback option is determined from multiple playback options and the target video segment corresponding to the target playback option is played.

10. The method according to any one of claims 1 to 9, characterized in that, The method further includes: Get the filtering information; Obtain the second detection command; In response to the second detection command, a description of the second target object in the images included in the target video segment is displayed. The target video segment consists of multiple frames of images containing the first target object in the monitoring video data. The filtering information is used to filter the second target object from the first target object.

11. The method according to any one of claims 1 to 10, characterized in that, The surveillance video data is from multiple channels, and the method further includes: Get the selection command; In response to the selection instruction, the target surveillance video data is determined from the multi-channel surveillance video data; The step of displaying descriptive information of the first target object in the image included in the surveillance video data in response to the first detection command includes: In response to the first detection command, the description information of the first target object in the image included in the target surveillance video data is displayed.

12. The method according to any one of claims 1 to 10, characterized in that, The method further includes: Get the time setting command; In response to the time setting command, a detection time period is determined; The step of displaying descriptive information of the first target object in the image included in the surveillance video data in response to the first detection command includes: In response to the first detection command, the description information of the first target object in the images included in the surveillance video data located within the detection time period is displayed.

13. The method according to any one of claims 1 to 10, characterized in that, The method further includes: In response to the first detection command, the image included in the surveillance video data is displayed and the first target object is highlighted.

14. The method according to any one of claims 1 to 13, characterized in that, The method further includes: Acquire other sensor data associated with the surveillance video data, which is used to assist in generating the descriptive information.

15. The method according to any one of claims 1 to 14, characterized in that, The description information is displayed in a preset format.

16. The method according to claim 15, characterized in that, The preset format can be set by the user or by the system.

17. A monitoring system, characterized in that, The monitoring system includes cameras and edge computing devices, wherein: The camera is used to collect surveillance video data, which includes multiple frames of images. The edge computing device is configured to acquire the surveillance video data; acquire a first detection command; and, in response to the first detection command, display descriptive information of a first target object in the image included in the surveillance video data.

18. The monitoring system according to claim 17, characterized in that, The camera may be one or more, and one camera may be used to collect one or more channels of surveillance video data.

19. The monitoring system according to claim 17 or 18, characterized in that, The edge computing device may be one or more; when there are multiple edge computing devices, the multiple edge computing devices are used to collaboratively generate descriptive information of the first target object in the image included in the surveillance video data.

20. An edge computing device, characterized in that, The edge computing device is used for Acquire surveillance video data collected by a camera, wherein the surveillance video data includes multiple frames of images; Obtain the first detection command; In response to the first detection command, the description information of the first target object in the image included in the monitoring video data is displayed.

21. A camera, characterized in that, The camera is used for Collect surveillance video data, which includes multiple frames of images; Obtain the first detection command; In response to the first detection command, the description information of the first target object in the image included in the monitoring video data is displayed.

22. An electronic device, characterized in that, include: A memory and a processor, wherein the memory is coupled to the processor; The memory stores program instructions that, when executed by the processor, cause the electronic device to perform the method as described in any one of claims 1 to 16.

23. A chip, characterized in that, It includes one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the method as described in any one of claims 1 to 16 are performed.

24. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed on a computer or processor, causes the computer or processor to perform the method as described in any one of claims 1 to 16.

25. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed by a computer or processor, cause the steps of the method as described in any one of claims 1 to 16 to be performed.

Images