Target event detection method and device, storage medium and electronic device

Abstract

Embodiments of the present application provide a target event detection method and device, a storage medium and an electronic device, wherein the method comprises: obtaining N object groups by performing object detection on each of N image frames, wherein N is a value determined according to a preset time length corresponding to the target event; determining M object groups in the N object groups that satisfy a target condition, each of the M object groups comprising a target object set; when the ratio of M to N is greater than a predetermined proportion threshold, determining whether the number of objects of each category in the target object set is greater than or equal to a preset number threshold corresponding to the target event; and when it is determined that the number of objects of each category is greater than or equal to the preset number threshold, determining that the target event is detected in the N image frames. Through the embodiments of the present application, the problem of low detection efficiency of whether a target event occurs in an image frame in the related art is solved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically, to a method, apparatus, storage medium, and electronic device for detecting target events. Background Technology

[0002] Video surveillance technology has been widely applied in many fields, and video analysis and processing has become a research hotspot. In related technologies, video analysis and processing mainly focus on the location detection and attribute labeling of image or video targets. For example, detecting the types of objects appearing in image frames, using target tracking algorithms to track targets to be labeled in the video, and using the tracking results obtained by the algorithm to label the targets to be labeled in the video. It can be seen that the analysis and processing of video or image frames in related technologies focuses on target detection and automatic labeling, but does not involve whether the target has actually performed any events. That is, the detection efficiency of whether target events have occurred in video or image frames in related technologies is relatively low.

[0003] There is currently no effective solution to the problem of low detection efficiency of target events in image frames in related technologies. Summary of the Invention

[0004] The present invention provides a method, apparatus, storage medium and electronic device for detecting target events, so as to at least solve the problem of low detection efficiency of whether a target event has occurred in an image frame in the related art.

[0005] According to an embodiment of the present invention, a method for detecting a target event is provided, comprising: performing object detection on each of N image frames to obtain N object groups, wherein each image frame corresponds to one object group, N is a positive integer greater than or equal to 1, and N is a value determined according to a preset duration corresponding to a target event; determining M object groups among the N object groups that satisfy the target conditions, wherein each of the M object groups includes a target object set, and M is a positive integer greater than or equal to 1 and less than or equal to N; if the ratio of M to N is greater than a predetermined ratio threshold, determining whether the number of each type of object included in the target object set is greater than or equal to a preset number threshold corresponding to the target event, wherein each type of object corresponds to a preset number threshold; if it is determined that the number of each type of object included in the target object set is greater than or equal to the preset number threshold, determining that the target event is detected in the N image frames.

[0006] In an exemplary embodiment, determining whether the number of objects of each category included in the target object set is greater than or equal to a preset quantity threshold corresponding to the target event includes: determining K categories of objects included in the target object set, and determining the number J of objects of category i among the K categories of objects. [i] Where K is a positive integer greater than or equal to 1, and i is a positive integer greater than or equal to 1 and less than or equal to K; compare K with the preset number of categories P to obtain a first comparison result, and compare the number J of objects in category i. [i] The preset number threshold Q of objects of category i in the P preset categories [i] The second comparison result is obtained, wherein the preset number of categories P is the preset number of object categories of the object when the target event occurs, and Q [i] P represents the number of objects of category i when the target event occurs, where P is a positive integer greater than or equal to 1, and the preset quantity threshold includes a preset quantity threshold Q for objects of category i. [i] Based on the first comparison result and the second comparison result, determine whether the number of objects of each category included in the target object set is greater than or equal to the preset quantity threshold corresponding to the target event.

[0007] In an exemplary embodiment, determining whether the number of objects in each category included in the target object set is greater than or equal to a preset number threshold corresponding to the target event based on the first comparison result and the second comparison result includes: where the first comparison result indicates that K is greater than or equal to P, and the second comparison result indicates that J satisfies J for each of the K categories. [i] Greater than or equal to Q [i] In the case where the number of objects of each category included in the target object set is greater than or equal to the preset quantity threshold, it is determined that...

[0008] In an exemplary embodiment, after determining that the target event occurs in the N image frames, the method further includes: labeling the N image frames to obtain target labeling results, wherein the target labeling results are used to indicate that the target event occurs in the N image frames; forming the N image frames and the target labeling results into target training samples in a training sample set, wherein the training sample set is used to train a target neural network model, and the target neural network model is used to detect whether the target event exists in the image frames input to the target neural network model.

[0009] In an exemplary embodiment, the step of annotating the N image frames to obtain target annotation results includes: annotating the target regions in the N image frames where the target event occurs, and annotating the event types of the target events, wherein the target annotation results include the target regions and the event types.

[0010] In one exemplary embodiment, the method further includes: determining that the target event was not detected in the N image frames if the ratio of M to N is less than or equal to the predetermined ratio threshold.

[0011] In an exemplary embodiment, before performing object detection on each of the N image frames to obtain N object groups, the method further includes: detecting the current image frame to obtain the current object group; and if the number of objects of each category included in the current object group is greater than or equal to the preset number threshold, acquiring the N image frames, wherein the N image frames include a group of image frames including the current image frame, and the maximum time interval between each image frame in the group of image frames is greater than or equal to the preset duration.

[0012] According to another embodiment of the present invention, a target event detection device is also provided, comprising: a first detection module, configured to perform object detection on each of N image frames to obtain N object groups, wherein each image frame corresponds to one object group, N is a positive integer greater than or equal to 1, and N is a value determined according to a preset duration corresponding to a target event; a first determination module, configured to determine M object groups among the N object groups that satisfy the target conditions, wherein each of the M object groups includes a target object set, and M is a positive integer greater than or equal to 1 and less than or equal to N; a second determination module, configured to determine whether the number of each type of object included in the target object set is greater than or equal to a preset number threshold corresponding to the target event when the ratio of M to N is greater than a predetermined ratio threshold, wherein each type of object corresponds to a preset number threshold; and a third determination module, configured to determine that the target event is detected in the N image frames when it is determined that the number of each type of object included in the target object set is greater than or equal to the preset number threshold.

[0013] According to yet another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0014] According to yet another embodiment of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0015] This invention achieves the goal of detecting objects in each of N image frames to obtain N object groups, where each image frame corresponds to one object group, and N is a value determined based on a preset duration corresponding to the target event. M object groups satisfying the target conditions are identified from the N object groups, where each of the M object groups includes a set of target objects. If the ratio of M to N is greater than a predetermined ratio threshold, it is determined whether the number of objects of each category in the target object set is greater than or equal to a preset quantity threshold corresponding to the target event, where each object category corresponds to a preset quantity threshold. If the number of objects of each category in the target object set is determined to be greater than or equal to the preset quantity threshold, it is determined that a target event has been detected in the N image frames. This achieves the goal of detecting whether a target event behavior has substantially occurred in a video or image frame, avoiding the problem in related technologies that mainly detect the type and location of objects appearing in the image frame, but cannot accurately detect whether a target event has occurred in the video or image frame. Therefore, this solves the problem of low detection efficiency for whether a target event has occurred in an image frame in related technologies, and achieves the effect of improving the efficiency of detecting whether a target event has occurred in an image frame. Attached Figure Description

[0016] Figure 1 This is a block diagram of the mobile terminal hardware structure of the target event detection method according to an embodiment of the present invention;

[0017] Figure 2 This is a flowchart of a target event detection method according to an embodiment of the present invention;

[0018] Figure 3 This is a structural diagram of a data annotation device according to a specific embodiment of the present invention;

[0019] Figure 4 This is a data annotation flowchart according to a specific embodiment of the present invention;

[0020] Figure 5 This is a flowchart of event behavior analysis according to a specific embodiment of the present invention;

[0021] Figure 6 This is an example diagram of event behavior annotation according to a specific embodiment of the present invention;

[0022] Figure 7 This is a structural block diagram of a target event detection device according to an embodiment of the present invention. Detailed Implementation

[0023] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings and examples.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0025] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a block diagram of the mobile terminal hardware structure of the target event detection method according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0026] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the target event detection method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0027] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0028] This embodiment provides a method for detecting target events. Figure 2 This is a flowchart of a target event detection method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0029] Step S202: Perform object detection on each of the N image frames to obtain N object groups, where each image frame corresponds to one object group, N is a positive integer greater than or equal to 1, and N is a value determined according to the preset duration corresponding to the target event.

[0030] Step S204: Determine M object groups that satisfy the target conditions from the N object groups, wherein each of the M object groups includes a target object set, and M is a positive integer greater than or equal to 1 and less than or equal to N;

[0031] Step S206: If the ratio of M to N is greater than a predetermined ratio threshold, determine whether the number of objects of each category included in the target object set is greater than or equal to a preset number threshold corresponding to the target event, wherein each category of objects corresponds to a preset number threshold.

[0032] Step S208: If it is determined that the number of objects of each category included in the target object set is greater than or equal to the preset number threshold, the target event is detected in the N image frames.

[0033] Through the above steps, object detection is performed on each of the N image frames to obtain N object groups, where each image frame corresponds to one object group, and N is a value determined based on a preset duration corresponding to the target event. M object groups that satisfy the target conditions are identified from the N object groups, where each of the M object groups includes a set of target objects. If the ratio of M to N is greater than a predetermined ratio threshold, it is determined whether the number of objects of each category included in the target object set is greater than or equal to a preset quantity threshold corresponding to the target event, where each category of objects corresponds to a preset quantity threshold. If the number of objects of each category included in the target object set is determined to be greater than or equal to the preset quantity threshold, it is determined that a target event has been detected in the N image frames. This achieves the purpose of detecting whether a target event behavior has substantially occurred in a video or image frame, avoiding the problem in related technologies that mainly detect the type and location of objects appearing in the image frame, but cannot accurately detect whether a target event has occurred in the video or image frame. Therefore, this solves the problem of low detection efficiency for whether a target event has occurred in an image frame in related technologies, and achieves the effect of improving the efficiency of detecting whether a target event has occurred in an image frame.

[0034] The entity performing the above steps may be an image processing system, an image processor, a video surveillance system, a terminal, or a processor with human-computer interaction capabilities configured on a storage device, or a processing device or processing unit with similar processing capabilities, but is not limited to these.

[0035] In the above embodiments, object detection is performed on each of the N image frames to obtain N object groups. Each image frame corresponds to one object group, meaning that object detection on one image frame yields one object group. Each object group includes one or more objects. If objects in an image frame are occluded, it is possible that no objects will be found in the resulting object group. N is a value determined based on a preset duration corresponding to the target event. For example, taking the target event as a pedestrian going onto an overpass, assuming the preset duration is 1 second, the target event is considered to have occurred if a pedestrian (e.g., pedestrian A) is detected continuously on the overpass for 1 second. N can be 25 (or 30, or other values), meaning that for 25 consecutive frames (or 3... Object detection is performed on 0 frames (or other frame numbers). Optionally, frames can be extracted or captured from a video segment (e.g., a 1-second video or a 2-second video) to obtain 10 frames (or other frame numbers). Object detection is then performed on these 10 frames. The detection result for each frame yields an object group. Thus, object detection on N image frames yields N object groups, and each object group can include one or more objects. Each object group can include people and / or vehicles and / or mobile phones or other objects. For example, when the target event to be detected is a driving and making a phone call event, the image frames can be detected to see if they include people, vehicles, and mobile phones. M object groups that meet the target conditions are determined from the N object groups, where M objects... Each object in a group includes the same set of target objects. For example, taking the target event as driving and making a phone call, the target object set could be a set of people, vehicles, and mobile phones. Taking the 1-second video containing 25 frames (i.e., N=25) as an example, object detection on these 25 frames yields 25 object groups, meaning one object group is detected in each frame. Then, from these 25 object groups, M object groups are identified that all include the same set of target objects (e.g., people, vehicles, and mobile phones). In other words, M frames all contain the same set of target objects. If the ratio of M to N is greater than a predetermined threshold, then each set of target objects is determined to contain the same set of target objects. Whether the number of objects of each category is greater than or equal to the preset number threshold corresponding to the target event. For example, taking the above-mentioned preset ratio threshold as 80% (or other values) as an example, when N=25, when M=22 is determined, the ratio of M to N is greater than 80%. At this time, whether the number of objects included in the target object set is greater than or equal to the preset number corresponding to the target event. For example, the preset number threshold corresponding to the target event (such as the above-mentioned driving and making a phone call event) is the number of people = 1, the number of vehicles = 1, and the number of mobile phones = 1. At this time, whether the number of each type of object in the above-mentioned target object set is greater than or equal to the number of each type of object corresponding to the target event (i.e., the above-mentioned preset number threshold);If the number of objects of each category in the target object set is greater than or equal to a preset threshold, a target event is detected in N image frames. When the number of each type of object in the target object set is greater than or equal to a preset number, a target event is determined to have occurred in the N image frames, i.e., a driving and talking on the phone event has occurred. If the target event is a traffic congestion event, the target object set can include multiple (e.g., 10, or other numbers) vehicles. That is, from the N object groups, M object groups are determined, each containing the same 10 vehicles. Then, following the same steps as above, it is determined whether the ratio is greater than a predetermined ratio threshold. If the predetermined ratio threshold is met, it is further determined whether the number of objects (e.g., the number of vehicles) in the target object set is greater than or equal to the preset threshold corresponding to the target event. Assuming the preset number of vehicles (i.e., the preset threshold) required for a traffic congestion event is 8, it can be determined that a target event (i.e., a traffic congestion event) exists in the N image frames. It should be noted that the aforementioned predetermined ratio threshold can be set according to different application scenarios, and the preset duration and preset quantity thresholds corresponding to the target event can also be set according to different application scenarios (such as target event detection in different situations). This embodiment achieves the purpose of detecting whether a target event behavior has substantially occurred in a video or image frame, avoiding the problem in related technologies that mainly detect the type and location of objects appearing in the image frame, which cannot accurately determine whether a target event has occurred in the video or image frame. Therefore, it solves the problem of low efficiency in determining whether a target event has occurred in an image frame in related technologies, achieving the effect of improving the efficiency of determining whether a target event has occurred in an image frame.

[0036] In an optional embodiment, determining whether the number of objects of each category included in the target object set is greater than or equal to a preset quantity threshold corresponding to the target event includes: determining K categories of objects included in the target object set, and determining the number J of objects of category i among the K categories of objects. [i] Where K is a positive integer greater than or equal to 1, and i is a positive integer greater than or equal to 1 and less than or equal to K; compare K with the preset number of categories P to obtain a first comparison result, and compare the number J of objects in category i. [i] The preset number threshold Q of objects of category i in the P preset categories [i] The second comparison result is obtained, wherein the preset number of categories P is the preset number of object categories of the object when the target event occurs, and Q [i] P represents the number of objects of category i when the target event occurs, where P is a positive integer greater than or equal to 1, and the preset quantity threshold includes a preset quantity threshold Q for objects of category i. [i]Based on the first comparison result and the second comparison result, determine whether the number of objects of each category included in the target object set is greater than or equal to the preset quantity threshold corresponding to the target event. In this embodiment, determine K categories of objects included in the target object set, for example, the target object set includes objects of different categories such as people, vehicles, and mobile phones, and determine the number of objects in each of the K categories, for example, determine the number J of objects of category i in the K categories. [i] This yields K quantities, such as J. [1] J [2] , ... J [K] Suppose the target event requires a preset number of object categories, P, and a threshold number for each of the P categories. For example, the preset threshold number of objects of category i in the P categories is Q. [i] This yields a total of P preset quantity thresholds. It should be noted that J... [i] With Q [i] This represents the number of objects of the same category i. For example, the "driving and making a phone call" event requires three preset object categories (i.e., P=3), such as people, vehicles, and mobile phones. The required quantity thresholds for each object category are: 1 for people, 1 for vehicles, and 1 for mobile phones. Therefore, the preset quantity for the "driving and making a phone call" event includes: 3 object categories, with 1, 1, and 1 objects in each of the three categories, respectively. K is then compared with the preset category quantity P to obtain a first comparison result, and K quantities are compared with P preset quantity thresholds to obtain a second comparison result. Based on the first and second comparison results, it is determined whether the number of objects in the target object set is greater than or equal to the preset quantity threshold corresponding to the target event. Through this embodiment, by determining the number of object categories included in the target object set and the number of objects in each category, the purpose of determining whether the number of objects in the target object set is greater than or equal to the preset quantity threshold corresponding to the target event is achieved.

[0037] In an optional embodiment, determining whether the number of objects of each category included in the target object set is greater than or equal to a preset number threshold corresponding to the target event based on the first comparison result and the second comparison result includes: when the first comparison result indicates that K is greater than or equal to P, and the second comparison result indicates that J satisfies J for each of the K categories. [i] Greater than or equal to Q [i] In this case, it is determined that the number of objects in each category included in the target object set is greater than or equal to the preset number threshold. In this embodiment, when K is greater than or equal to P, and the number of objects J in each of the K categories is greater than or equal to P, the target object set is determined to be greater than or equal to P. [i]Greater than or equal to Q [i] When, that is, for any category, J satisfies [i] Greater than or equal to Q [i] For example, for category s, satisfying J [s] Greater than or equal to Q [s] At this point, it can be determined that the number of objects in each category included in the target object set is greater than or equal to a preset quantity threshold. For example, if the P quantities are: number of people = 1, number of vehicles = 1, and number of mobile phones = 1, then only when the number of people corresponding to the K quantities is ≥ 1, the number of vehicles corresponding to the K quantities is ≥ 1, and the number of mobile phones corresponding to the K quantities is ≥ 1, can it be determined that the number of objects in each category included in the target object set is greater than or equal to the preset quantity threshold. When K is greater than P, for example, K categories include category r, but the P preset categories may not include category r, this is equivalent to Q. [r] =0; Taking a crowd gathering event as an example, the preset number of categories P = 1 (e.g., category is people). For example, the preset number of objects corresponding to the category of people is 10 (or other values), i.e., Q. [i] =10, where i represents the category of people, meaning that the number of people appearing in the same frame is 10. When the duration reaches the preset duration (such as 3 seconds, or other durations), it can be considered that a gathering of people has occurred. Optionally, in practical applications, for the above-mentioned driving and talking on the phone event, the positional relationship and duration of different target objects can be further determined. For example, when performing object detection on each frame of a video, the type of object and the positional information of the object can be detected. The detection results can also include the acquisition time of the image frame, etc. In this way, the positional relationship between different target objects can be determined, such as the positional relationship between people, vehicles and mobile phones. This can further improve the accuracy of determining target events.

[0038] In an optional embodiment, after determining that the target event occurs in the N image frames, the method further includes: labeling the N image frames to obtain target labeling results, wherein the target labeling results are used to indicate that the target event occurs in the N image frames; forming the N image frames and the target labeling results into target training samples in a training sample set, wherein the training sample set is used to train a target neural network model, and the target neural network model is used to detect whether the target event exists in the image frames input to the target neural network model. In this embodiment, after determining that a target event occurs in N image frames, the N image frames can be further annotated to obtain target annotation results. For example, the target annotation results may include the area where the target event occurs and / or the type of target event, such as a driving and talking on the phone event, a traffic congestion event, or a crowd gathering event. Then, the N image frames and the target annotation results can be used to form target training samples. Following the same method as in this embodiment, it can be determined whether other types of target events exist in image frames or videos in other different situations. After determining whether target events occur or not in image frames or videos of the same situation at different times or under different environmental conditions, or after determining whether a certain type of event occurs or not in image frames or videos of different situations, the relevant image frames or videos can also be annotated. This way, many training samples can be obtained to form a training sample set. This training sample set can be used to train a target neural network model. The neural network model that has been trained to meet the requirements can be used to detect image frames or videos to detect whether target events exist. Through this embodiment, the purpose of obtaining target training samples is achieved, and then the target neural network model can be trained using the training samples to obtain the corresponding model.

[0039] In an optional embodiment, the annotation of the N image frames to obtain the target annotation result includes: annotating the target region in the N image frames where the target event occurs, and annotating the event type of the target event, wherein the target annotation result includes the target region and the event type. In this embodiment, the target region in the N image frames where the target event occurs, such as an event frame, can be annotated, and the event type of the target event can be annotated. Optionally, the identifier number of the target event, such as the event ID, can also be annotated. In practical applications, when the N image frames or video are used for playback in a player, the corresponding time can be marked on the player's progress bar. For example, the event ID can be marked with a red arrow. Of course, multiple different types of events can also be identified within the same time period, and the detection of each event can be performed using the same method as described above. Optionally, when no event occurs during a certain time period of a video, the corresponding position in the progress bar can be grayed out, which facilitates later querying or searching for video segments where the event occurred. This provides convenience for users and improves the user experience.

[0040] In an optional embodiment, the method further includes: determining that the target event was not detected in the N image frames when the ratio of M to N is less than or equal to the predetermined ratio threshold. In this embodiment, when the ratio of M to N is less than or equal to the predetermined ratio threshold, it can be determined that no target event occurred in the N image frames. For example, taking N=25 as an example, when it is determined that 3 groups (or 10 groups, or other groups) out of 25 groups of objects meet the target condition, that is, only 3 groups of objects all include the same set of target objects, the ratio of M to N is less than the predetermined ratio threshold (e.g., 80%), and it can be determined that no target event was detected in the N image frames.

[0041] In an optional embodiment, before performing object detection on each of the N image frames to obtain N detection result object groups, the method further includes: detecting the current image frame to obtain a current detection result object group, wherein the current detection result includes a current group of objects contained in the current image frame, and the current group of objects includes one or more objects; if the number of objects of each category included in the current group of objects is greater than or equal to the preset number threshold, obtaining the N image frames, wherein the N image frames include a group of image frames including the current image frame, and the maximum time interval between each image frame in the group of image frames is greater than or equal to the preset duration. In this embodiment, the current image frame can be detected to obtain a current detection result object group. When the current detection result determines that the number of objects of each category included in the current object group is greater than or equal to a preset number, N image frames are then acquired. The N image frames can be a group of image frames including the current image frame. For example, it can be a group of image frames with a predetermined duration starting from the current image frame. Optionally, a group of image frames can be multiple consecutive image frames, or it can be multiple image frames obtained by acquiring or extracting frames from a continuous segment of image frames. For example, it can be 25 consecutive image frames included in a 1-second video, or it can be multiple image frames obtained by acquiring or extracting frames from 25 consecutive image frames. The system obtains 10 image frames by sampling or frame extraction; the time interval between the last image frame and the first image frame (such as the current image frame) in a group of image frames is greater than or equal to the preset duration (such as 1s, 3s, or others); through this embodiment, if the number of objects included in the current object group detected by the current image frame does not meet the preset number, it is not necessary to obtain N image frames. At this time, the next image frame can be detected. When the detected current object group (such as the aforementioned target object set) meets the preset number threshold of each type of object corresponding to the target event, N image frames are obtained to determine whether the continuous occurrence time of the target object set meets the preset duration requirement of the target event.

[0042] Obviously, the embodiments described above are only some embodiments of the present invention, and not all embodiments. The present invention will be specifically described below with reference to the embodiments.

[0043] Figure 3 This is a structural diagram of a data annotation device according to a specific embodiment of the present invention. The device includes: a data preprocessing module, an event behavior analysis module, a data processing module, a manual confirmation module, and a model data training module.

[0044] Figure 4 This is a data annotation flowchart according to a specific embodiment of the present invention, the process including:

[0045] S402, acquire video data or continuous image frame data (corresponding to the aforementioned N image frames);

[0046] S404 uses a built-in general target detection and tracking model to detect and track data;

[0047] S406, combined with the time in the data source, classifies and clusters the detected targets;

[0048] S408, determine whether any events meeting the conditions have occurred within the corresponding time period and classify and process them accordingly;

[0049] S410, if the judgment result of step S408 is negative, the data that does not meet the requirements is grayed out, and then the process proceeds to step S420.

[0050] S412, if the judgment result of step S408 is yes, label the data that meets the conditions with event behavior boxes and corresponding event types;

[0051] S414, mark and remind users of the time period in which the event occurred;

[0052] S416, Confirm automatic annotation data;

[0053] S418, training data, to obtain the corresponding model;

[0054] S420, End.

[0055] The functions of each module in the above device and the above process are explained as follows:

[0056] (1) Data preprocessing module

[0057] This module mainly consists of two parts:

[0058] A. Obtaining source data and source data information: Source data can be initial video data or continuous image frame data. Source data information includes the source data scene and the corresponding event behavior tags within that scene;

[0059] B. Preprocessing data: 1) Based on the input source data scenario, load the general target detection and tracking model for the corresponding scenario;

[0060] 2) Use the loaded model to perform target detection and tracking on the data source;

[0061] 3) Classify and cluster the detected and tracked targets. Classification: Differentiate targets of the same type. Clustering: Archive tracked targets or targets with high similarity.

[0062] The functions and roles of the data preprocessing module are as follows: Figure 4 Steps S402-S406 in the process.

[0063] (2) Event Behavior Analysis Module

[0064] Definition of event behavior: Actions that occur and are executed effectively within a unit of time.

[0065] The biggest difference between event behavior analysis and intelligent target detection is that target detection only detects targets, while event behavior analysis needs to analyze actions within a unit of time, which can be performed by one or more objects in combination. Therefore, event behavior is classified from the perspective of individual event performers to improve the accuracy of event behavior analysis.

[0066] Table 1 provides definitions and explanations for different event behavior types. This table is only one example and may include other event behavior types.

[0067] Table 1

[0068]

[0069]

[0070] The corresponding labels configured for each scenario are shown in Table 2. The following example uses a highway road scenario.

[0071] Table 2

[0072]

[0073] The following formula is derived from the data preprocessing module (detection, tracking + classification, clustering) (taking the results data per unit time as an example):

[0074]

[0075] in,

[0076] m k This indicates the number of target types in the current frame that match label k (corresponding to the aforementioned K categories);

[0077] M k min This represents the number of target categories required for the event behavior (label k) (corresponding to the aforementioned preset number of categories P);

[0078] n k This indicates the number of targets in the current frame that match label k under each target category (corresponding to the aforementioned K quantities);

[0079] N k min This represents the number of targets under each target category required for the event behavior (label k) (corresponding to the aforementioned P quantities);

[0080] t represents the duration of the detected target's presence;

[0081] T k min Indicates the minimum duration required for the event behavior (label k) (corresponding to the aforementioned preset duration);

[0082] The functions and roles of the event behavior analysis module are as follows: Figure 4 Steps S408-S412 in the process.

[0083] The following section will explain the overall analysis process of the event behavior analysis module. Figure 5 This is a flowchart of event behavior analysis according to a specific embodiment of the present invention. The process includes:

[0084] S502, obtain the target type (corresponding to the aforementioned K categories) and target quantity (corresponding to the aforementioned K quantities) in the current frame;

[0085] S504, match the target types required by the event label type in descending order of quantity;

[0086] S506, determine whether the target type (corresponding to the aforementioned preset number of categories P) matches;

[0087] S508, if the judgment result of the above step S506 is yes, match in descending order of the number of targets required to be included in the event tag type;

[0088] S510, determine whether the target number (corresponding to the aforementioned P quantities) matches;

[0089] S512, if the judgment result of the above step S510 is yes, determine whether the target duration meets the standard;

[0090] If the judgment result of step S506, or S510, or S512 is negative, proceed to step S514.

[0091] S514, continue matching until all event tags and all target time periods are matched;

[0092] S516, If the judgment result of step S512 is yes, automatically label the corresponding tag event behavior within this period of time.

[0093] S518, End.

[0094] (3) Data processing module

[0095] The main operations of this module are:

[0096] The detected targets are labeled with the most likely event behavior tags from the event behavior analysis module: event box, event type, and event ID. The labels are then added in the corresponding locations, such as... Figure 6 As shown, for example Figure 6 The arrow in the progress bar of the video.

[0097] The functions and roles of the data processing module are as follows: Figure 4 Step S414 in the process.

[0098] (4) Manual confirmation module

[0099] The automatically labeled data is pushed to the operator for confirmation. This module, aside from data push, is primarily performed manually and is not considered a protection point; therefore, it will not be discussed further.

[0100] This section corresponds to Figure 4 Step S416 in the process.

[0101] (5) Model Data Training Module

[0102] The inspection will use reasonable and compliant data to train the model and obtain the corresponding results.

[0103] This section corresponds to Figure 4 Step S418 in the process.

[0104] In the above embodiments, based on (1), according to the type of input data scenario, the corresponding model is selected for target detection, tracking, classification and clustering; based on (2), the target result in the above steps is calculated and matched with the event behavior label in the input scenario, the matched behavior event is labeled and the user is notified, so as to improve the efficiency of event behavior data labeling.

[0105] Through the embodiments of this application, when a target substantially performs an action in continuous image data or video data: 1) the video segment where the event occurred is indicated; 2) invalid data segments (which cannot form the conditions for the event to occur) are filtered out, and the target event occurrence segment data is automatically labeled; thus solving the technical problem of low labeling efficiency of target event behavior.

[0106] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0107] This embodiment also provides a target event detection device. Figure 7 This is a structural block diagram of a target event detection device according to an embodiment of the present invention, such as... Figure 7 As shown, the device includes:

[0108] The first detection module 702 is used to perform object detection on each of the N image frames to obtain N object groups, wherein each image frame corresponds to one object group, N is a positive integer greater than or equal to 1, and N is a value determined according to the preset duration corresponding to the target event.

[0109] The first determining module 704 is used to determine M object groups that satisfy the target conditions among the N object groups, wherein each of the M object groups includes a target object set, and M is a positive integer greater than or equal to 1 and less than or equal to N;

[0110] The second determining module 706 is used to determine whether the number of objects of each category included in the target object set is greater than or equal to a preset number threshold corresponding to the target event when the ratio of M to N is greater than a predetermined ratio threshold. Each category of objects corresponds to a preset number threshold.

[0111] The third determining module 708 is used to determine that the target event is detected in the N image frames when the number of objects of each category included in the target object set is greater than or equal to the preset number threshold.

[0112] In an optional embodiment, the second determining module 706 includes: a first determining unit, configured to determine K categories of objects included in the target object set, and determine the number J of objects of category i among the K categories of objects. [i] Where K is a positive integer greater than or equal to 1, and i is a positive integer greater than or equal to 1 and less than or equal to K; the comparison unit is used to compare K with a preset number of categories P to obtain a first comparison result, and to compare the number J of objects of category i.[i] The preset number threshold Q of objects of category i in the P preset categories [i] The second comparison result is obtained, wherein the preset number of categories P is the preset number of object categories of the object when the target event occurs, and Q [i] P represents the number of objects of category i when the target event occurs, where P is a positive integer greater than or equal to 1, and the preset quantity threshold includes a preset quantity threshold Q for objects of category i. [i] The second determining unit is used to determine, based on the first comparison result and the second comparison result, whether the number of objects of each category included in the target object set is greater than or equal to a preset number threshold corresponding to the target event.

[0113] In an optional embodiment, the second determining unit includes: a first determining subunit, configured to determine if the first comparison result indicates that K is greater than or equal to P, and the second comparison result indicates that J satisfies J for each of the K categories. [i] Greater than or equal to Q [i] In the case where the number of objects of each category included in the target object set is greater than or equal to the preset quantity threshold, it is determined that...

[0114] In an optional embodiment, the above apparatus further includes: an obtaining module, configured to annotate the N image frames after determining that the target event occurs in the N image frames to obtain a target annotation result, wherein the target annotation result is used to indicate that the target event occurs in the N image frames; and a forming module, configured to form the N image frames and the target annotation result into a target training sample in a training sample set, wherein the training sample set is used to train a target neural network model, and the target neural network model is used to detect whether the target event exists in the image frames input to the target neural network model.

[0115] In an optional embodiment, the above-mentioned obtaining module includes: a labeling unit, used to label the target region in the N image frames where the target event occurs, and to label the event type of the target event, wherein the target labeling result includes the target region and the event type.

[0116] In an optional embodiment, the above apparatus further includes: a fourth determining module, configured to determine that the target event was not detected in the N image frames if the ratio of M to N is less than or equal to the predetermined ratio threshold.

[0117] In an optional embodiment, the above apparatus further includes: a second detection module, configured to detect the current image frame to obtain a current object group; and an acquisition module, configured to acquire the N image frames when the number of objects of each category included in the current object group is greater than or equal to the preset number threshold, wherein the N image frames include a group of image frames including the current image frame, and the maximum time interval between each image frame in the group of image frames is greater than or equal to the preset duration.

[0118] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0119] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0120] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0121] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0122] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0123] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0124] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0125] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting a target event, characterized in that, include: Object detection is performed on each of the N image frames to obtain N object groups, where each image frame corresponds to one object group, and N is a positive integer greater than or equal to 1, which is a value determined according to the preset duration corresponding to the target event. Determine M object groups from the N object groups that satisfy the target conditions, wherein each of the M object groups includes the same set of target objects, and M is a positive integer greater than or equal to 1 and less than or equal to N; If the ratio of M to N is greater than a predetermined ratio threshold, determine whether the number of objects of each category included in the target object set is greater than or equal to a preset number threshold corresponding to the target event, wherein each category of objects corresponds to a preset number threshold; If the number of objects of each category included in the target object set is greater than or equal to the preset number threshold, the target event is determined to have been detected in the N image frames.

2. The method according to claim 1, characterized in that, Determining whether the number of objects of each category included in the target object set is greater than or equal to a preset quantity threshold corresponding to the target event includes: Determine K categories of objects included in the target object set, and determine J, the number of objects of category i among the K categories. [i] , where K is a positive integer greater than or equal to 1, and i is a positive integer greater than or equal to 1 and less than or equal to K; Compare K with the preset number of categories P to obtain the first comparison result, and then compare the number J of objects in category i. [i] The preset number threshold Q of objects of category i in the P preset categories [i] The second comparison result is obtained, wherein the preset number of categories P is the preset number of object categories of the object when the target event occurs, and Q [i] P represents the number of objects of category i when the target event occurs, where P is a positive integer greater than or equal to 1, and the preset quantity threshold includes a preset quantity threshold Q for objects of category i. [i] ; Based on the first comparison result and the second comparison result, determine whether the number of objects of each category included in the target object set is greater than or equal to a preset number threshold corresponding to the target event.

3. The method according to claim 2, characterized in that, The step of determining whether the number of objects of each category included in the target object set is greater than or equal to a preset quantity threshold corresponding to the target event based on the first comparison result and the second comparison result includes: The first comparison result indicates that K is greater than or equal to P, and the second comparison result indicates that J satisfies J for each of the K categories. [i] Greater than or equal to Q [i] In the case where the number of objects of each category included in the target object set is greater than or equal to the preset quantity threshold, it is determined that...

4. The method according to any one of claims 1 to 3, characterized in that, After determining that the target event occurs in the N image frames, the method further includes: The N image frames are labeled to obtain target labeling results, wherein the target labeling results are used to indicate that the target event occurs in the N image frames; The N image frames and the target annotation results are used to form the target training samples in the training sample set. The training sample set is used to train the target neural network model, and the target neural network model is used to detect whether the target event exists in the image frames input to the target neural network model.

5. The method according to claim 4, characterized in that, The step of annotating the N image frames to obtain the target annotation result includes: The target regions in the N image frames where the target event occurs are labeled, and the event types of the target events are labeled, wherein the target labeling results include the target regions and the event types.

6. The method according to any one of claims 1 to 3, characterized in that, The method further includes: If the ratio of M to N is less than or equal to the predetermined ratio threshold, it is determined that the target event was not detected in the N image frames.

7. The method according to any one of claims 1 to 3, characterized in that, Before performing object detection on each of the N image frames to obtain N object groups, the method further includes: Detect the current image frame to obtain the current object group; If the number of objects of each category included in the current object group is greater than or equal to the preset number threshold, the N image frames are obtained, wherein the N image frames include a group of image frames including the current image frame, and the time interval between the last image frame and the first image frame in the group of image frames is greater than or equal to the preset duration.

8. A device for detecting a target event, characterized in that, include: The first detection module is used to perform object detection on each of the N image frames to obtain N object groups, where each image frame corresponds to one object group, N is a positive integer greater than or equal to 1, and N is a value determined according to the preset duration corresponding to the target event. The first determining module is used to determine M object groups that satisfy the target conditions among the N object groups, wherein each of the M object groups includes the same set of target objects, and M is a positive integer greater than or equal to 1 and less than or equal to N; The second determining module is used to determine whether the number of objects of each category included in the target object set is greater than or equal to a preset number threshold corresponding to the target event when the ratio of M to N is greater than a predetermined ratio threshold. Each category of objects corresponds to a preset number threshold. The third determining module is used to determine that the target event was detected in the N image frames if the number of objects of each category included in the target object set is greater than or equal to the preset number threshold.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 7.

Images