Video processing method and device thereof

Abstract

The present application discloses a video processing method and device thereof, and relates to the field of image processing technology. The method comprises: identifying video frames in a first video sequence, obtaining an abnormal behavior event of a specified type in the first video sequence and a target object related to the abnormal behavior event; obtaining a second video sequence according to the time when the abnormal behavior event occurs in the target object in the first video sequence; clustering the video frames in the second video sequence, obtaining at least one video subsequence associated with the target object; determining the correlation between each video subsequence and the target object and the abnormal behavior event, and selecting a target video subsequence from the video subsequence according to the correlation between each video subsequence; splicing the target video subsequence to obtain a target video sequence associated with the target object and the abnormal behavior event.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a video processing method and device thereof. Background Art

[0002] With the development of Internet technology, image acquisition equipment is widely distributed in various places such as hospitals, communities, and streets. Once an emergency or police situation occurs, the video (video sequence) related to the target object (which can be a person or an object) can be determined based on the images or videos collected by the image acquisition equipment, and corresponding measures can be taken in time.

[0003] However, with the increasing number of image acquisition devices, the size of video data generated every day is growing exponentially. If you want to determine the abnormal behavior information of the target object, you need to obtain the video related to the target object from the massive video data, which is time-consuming, labor-intensive and inefficient.

[0004] Therefore, how to efficiently process videos related to target objects from massive video data is a problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a video processing method for efficiently processing videos related to a target object.

[0006] In a first aspect, a video processing method is provided, comprising:

[0007] The video frames in a first video sequence are identified to obtain abnormal behavior events of a specified type in the first video sequence and a target object related to the abnormal behavior event; a second video sequence is generated according to the time when the abnormal behavior event occurs to the target object in the first video sequence, wherein the second video sequence includes a video sequence when the target object occurs the abnormal behavior, and a video sequence of a set time before and / or after the target object occurs the abnormal behavior; the video frames in the second video sequence are clustered to obtain at least one video subsequence associated with the target object; the correlation between the video subsequence and the target object and the abnormal behavior event is determined, and a target video subsequence is selected from the video subsequence according to the correlation between each video subsequence; the target video subsequences are spliced ​​to obtain a target video sequence associated with the target object and the abnormal behavior event.

[0008] Optionally, determining the relevance of each video subsequence in the at least one video subsequence associated with the target object to the target object and the abnormal behavior event includes:

[0009] The correlation between the first video subsequence and the target object and the abnormal behavior event is determined according to at least one of the time correlation, event main line correlation, and target object correlation of the first video subsequence, and the first video subsequence is any one of the video subsequences.

[0010] Optionally, the correlation between the first video subsequence, the target object and the abnormal behavior event satisfies the following formula:

[0011]

[0012] Among them, G represents the score corresponding to the correlation between the first video subsequence and the target object and the abnormal behavior event, d represents the score corresponding to the time correlation of the first video subsequence, e represents the score corresponding to the event main line correlation of the first video subsequence, and f represents the score corresponding to the target object correlation of the first video subsequence.

[0013] Optionally, the identifying of video frames in the first video sequence to obtain abnormal behavior events of a specified type in the first video sequence and target objects related to the abnormal behavior events includes:

[0014] The video frames in the first video sequence are identified to obtain the abnormal behavior event and the severity level corresponding to the abnormal behavior event; the correspondence between the pre-set event type and the attention level is queried according to the type of the abnormal behavior event to obtain the attention level corresponding to the abnormal behavior event; based on the severity level corresponding to the abnormal behavior event and the corresponding attention level, it is determined whether the abnormal behavior event meets the requirements; if the abnormal behavior event meets the requirements, the video frames containing the abnormal behavior event in the first video sequence are identified to obtain the target object related to the abnormal behavior event.

[0015] Optionally, determining whether the abnormal behavior event meets requirements according to the severity level and the corresponding attention level corresponding to the abnormal behavior event includes:

[0016] The score corresponding to the abnormal behavior event is calculated according to the following formula:

[0017] C=a*b

[0018] Among them, C is the score corresponding to the abnormal behavior event, a is the score corresponding to the severity level of the abnormal behavior event, and b is the score corresponding to the attention level of the abnormal behavior event; if the score corresponding to the abnormal behavior event is greater than the set threshold, it is determined that the abnormal behavior event meets the requirements.

[0019] Optionally, the splicing the target video subsequences includes:

[0020] The target video subsequences are spliced ​​in descending order of relevance.

[0021] In a second aspect, a video processing device is provided, comprising:

[0022] A determination module is used to identify video frames in a first video sequence to obtain abnormal behavior events of a specified type in the first video sequence and a target object related to the abnormal behavior event; an acquisition module is used to generate a second video sequence according to the time when the abnormal behavior event occurs to the target object in the first video sequence, the second video sequence including a video sequence when the target object occurs the abnormal behavior, and a video sequence of a set time before and / or after the target object occurs the abnormal behavior; a clustering module is used to cluster the video frames in the second video sequence to obtain at least one video subsequence associated with the target object; a target video subsequence screening module is used to determine the correlation between the video subsequence and the target object and the abnormal behavior event, and select a target video subsequence from the video subsequence according to the correlation of each video subsequence; a splicing module is used to splice the target video subsequences to obtain a target video sequence associated with the target object and the abnormal behavior event.

[0023] Optionally, the target video subsequence screening module is specifically used to:

[0024] The correlation between the first video subsequence and the target object and the abnormal behavior event is determined according to at least one of the time correlation, event main line correlation, and target object correlation of the first video subsequence, and the first video subsequence is any one of the video subsequences.

[0025] Optionally, the correlation between the first video subsequence, the target object and the abnormal behavior event satisfies the following formula:

[0026]

[0027] Among them, G represents the score corresponding to the correlation between the first video subsequence and the target object and the abnormal behavior event, d represents the score corresponding to the time correlation of the first video subsequence, e represents the score corresponding to the event main line correlation of the first video subsequence, and f represents the score corresponding to the target object correlation of the first video subsequence.

[0028] Optionally, the determining module is specifically used to:

[0029] The video frames in the first video sequence are identified to obtain the abnormal behavior event and the severity level corresponding to the abnormal behavior event; the correspondence between the pre-set event type and the attention level is queried according to the type of the abnormal behavior event to obtain the attention level corresponding to the abnormal behavior event; based on the severity level corresponding to the abnormal behavior event and the corresponding attention level, it is determined whether the abnormal behavior event meets the requirements; if the abnormal behavior event meets the requirements, the video frames containing the abnormal behavior event in the first video sequence are identified to obtain the target object related to the abnormal behavior event.

[0030] Optionally, the determining module is specifically used to:

[0031] The score corresponding to the abnormal behavior event is calculated according to the following formula:

[0032] C=a*b

[0033] Among them, C is the score corresponding to the abnormal behavior event, a is the score corresponding to the severity level of the abnormal behavior event, and b is the score corresponding to the attention level of the abnormal behavior event; if the score corresponding to the abnormal behavior event is greater than the set threshold, it is determined that the abnormal behavior event meets the requirements.

[0034] Optionally, the splicing module is specifically used for:

[0035] The target video subsequences are spliced ​​in descending order of relevance.

[0036] According to a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of the first aspects is implemented.

[0037] According to a fourth aspect, a computer-readable storage medium is provided, which stores a computer program executable by a computer device, and when the program is run on the computer device, the computer device executes the method described in any one of the first aspects.

[0038] In a fifth aspect, a computer program product is provided, the computer program product comprising a computer program stored on a computer-readable storage medium, the computer program comprising program instructions, and when the program instructions are executed by a computer, the computer is caused to execute any one of the methods described in the first aspect.

[0039] In an embodiment of the present application, video frames in a first video sequence are identified to obtain abnormal behavior events of a specified type in the first video sequence and target objects related to the abnormal behavior events, so as to highlight the key points and accurately identify the type of the abnormal behavior event and the related target objects; based on the time when the abnormal behavior event occurs in the target object in the first video sequence, a second video sequence is obtained, and the second video sequence is clustered to obtain at least one video subsequence associated with the target object, so as to focus on the trajectory of the target object and facilitate the improvement of the details of the abnormal behavior event; after determining the relevance of each video subsequence, and based on the relevance of each video subsequence, a target video subsequence is selected from the video subsequences associated with the target object, so as to delete unnecessary videos and shorten the video review time; the target video subsequences are spliced ​​to obtain a target video sequence associated with the target object and the abnormal behavior event, so as to focus on the result of the abnormal behavior event, thereby improving the efficiency of processing abnormal behavior events and related target objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of a system architecture provided for an embodiment of the present application;

[0041] Figure 2 A schematic diagram of a video processing method provided in an embodiment of the present application;

[0042] Figure 3 A schematic diagram of a process for determining a target object provided in an embodiment of the present application;

[0043] Figure 4 A video processing logic diagram provided for an embodiment of the present application;

[0044] Figure 5 A schematic diagram of the structure of a video processing device provided in an embodiment of the present application;

[0045] Figure 6 A schematic diagram of the structure of a communication device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0047] Some of the terms used in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0048] (1) In the embodiments of the present application, the terms "network" and "system" are often used interchangeably, but those skilled in the art can understand their meanings.

[0049] (2) In the embodiments of the present application, the term "plurality" refers to two or more than two, and other quantifiers are similar.

[0050] (3) “And / or” describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character “ / ” generally indicates that the associated objects are in an “or” relationship.

[0051] refer to Figure 1 , is a system architecture diagram applicable to an embodiment of the present application, the system architecture includes at least two parts: a processor unit 101 and a memory unit 102, wherein the processor unit 101 also includes at least a retrieval module 101a, a determination module 101b, an acquisition module 101c, an intelligent algorithm module 101d, and a screening module 101e, and the memory unit 102 includes at least a video storage module 102a, a video storage module 102b, and a video storage module 102c.

[0052] The retrieval module 101a can realize the video retrieval function.

[0053] The determination module 101b may identify the target object in the video sequence according to the video sequence retrieved by the retrieval module 101a.

[0054] The acquisition module 101c may be used to acquire a video sequence of the target object within a set time period.

[0055] The intelligent algorithm module 101d can perform abnormal behavior recognition, clustering and other processing on the target object according to the video sequence acquired by the acquisition module 101c, and split the video sequence into multiple sub-video sequences.

[0056] The screening module 101e can be used to perform secondary screening on the multiple sub-video sequences processed by the intelligent algorithm module 101d and splice them into a new video sequence.

[0057] The video storage module 102a can be used to store the video sequence retrieved by the retrieval module 101a; the video storage module 102b can be used to store the video sequence acquired by the acquisition module 101c; the video storage module 102a can be used to splice the video sequence into a new video sequence for storage.

[0058] In the embodiments of the present application, an abnormal behavior event may refer to an event that violates civilized norms or group behavior habits (such as running a red light, a car accident, speeding, fighting, etc.), and the abnormal behavior event may be predefined. The target object refers to the event subject related to the abnormal behavior event. For example, the target object of a car accident event may be the vehicle involved in the car accident, or the vehicle and person involved in the car accident. The target object of a fight event mainly includes the person involved in the fight.

[0059] It should be noted that the above system can be widely used in products with storage capabilities such as network hard disk recorders and digital hard disk recorders, and the embodiments of the present application do not limit this.

[0060] Taking the intelligent traffic scene as an example, the retrieval module 101a obtains a video sequence (for example, from 8 o'clock to 9 o'clock) from the image acquisition device, and sends the video sequence to the video storage module 102a and the determination module 101b; the determination module 101b identifies the video sequence sent by the retrieval module 101a, and determines that a car accident (a collision between a person and a car) occurred at 8:20-8:30 in the video sequence, as well as the target objects (vehicles, people) related to the car accident; the acquisition module 101c obtains the video sequence within the set time period (for example, 7:20-9:30) according to the time when the car accident occurred. The video sequence from 7:20 to 9:30 is sent to the intelligent algorithm module 101d. The intelligent algorithm module 101d clusters the target objects related to the car accident in the video sequence from 7:20 to 9:30 to form multiple sub-video sequences, which are sent to the screening module 101e and stored in the video storage module 102b. The screening module 101e screens out the video sub-sequences containing the car accident and the target objects related to the car accident from the multiple sub-video sequences, splices them together to obtain the target video sequence, and stores the target video sequence in the video storage module 102c.

[0061] based on Figure 1 The system architecture diagram shown in the figure, the embodiment of the present application provides a process of a video processing method, such as Figure 2 As shown, the process of the method can be performed by a video processing device (eg Figure 1 The processor unit 101 in the embodiment of the present invention is executed. The device can be implemented by software, hardware, or a combination of software and hardware. The process includes the following steps:

[0062] 201: Identify video frames in a first video sequence to obtain abnormal behavior events of a specified type in the first video sequence and target objects related to the abnormal behavior events.

[0063] Optionally, the identification of abnormal behavior events can be based on the type of abnormal behavior events, using a corresponding abnormal behavior identification algorithm, so that the type of abnormal behavior events in the first video can be accurately classified. Specifically, if it is a type of vehicle running a red light, based on the vehicle abnormal behavior identification algorithm (for example, Gaussian Mixed Hidden Markov Model (GM-HMM), target detection YOLO v3 and other algorithms), the abnormal behavior event of the vehicle running a red light and the target object related to the red light running can be obtained; if it is a type of people gathering and fighting, based on the abnormal behavior identification algorithm of the personnel (for example, Improve dense trajectory (Improve dense trajectory, iDT), two stream (TwoStream) algorithm, etc., the abnormal behavior event of people gathering and fighting and the target object related to the people gathering and fighting can be obtained.

[0064] Optionally, before identifying the video frames in the first video column, a video acquisition request may be received, which may include retrieval information such as time and place; the stored video data is queried according to the video acquisition request to obtain the requested first video sequence, and further, the first video sequence may be marked with a review label (for example, 20211202160623) for easy recording and query.

[0065] Optional, Figure 3 An implementation method of step 201 is shown as an example. Figure 3 As shown, a method process for determining a target object provided by an embodiment of the present application may include:

[0066] 301: Identify video frames in the first video sequence to obtain abnormal behavior events and severity levels corresponding to the abnormal behavior events.

[0067] Optionally, the severity level of the abnormal behavior event can be preset. Taking running a red light as an example, it is divided into four levels (very high, high, medium, and low). An abnormal behavior event with a very high severity level can be set as running a red light causing major personal and / or property losses, an abnormal behavior event with a high severity level can be set as running a red light causing serious personal and / or property losses, an abnormal behavior event with a medium severity level can be set as running a red light causing general personal and / or property losses, and an abnormal behavior event with a low severity level can be set as running a red light without causing personal and / or property losses. Specifically, different severity levels can correspond to different scores, as shown in Table 1.

[0068] Table 1: Severity rating rules for abnormal behavior events

[0069] Severity Points Extra high 4 high 3 middle 2 Low 1

[0070] 302: According to the type of the abnormal behavior event, a preset correspondence relationship between the event type and the attention degree is searched to obtain the attention degree corresponding to the abnormal behavior event.

[0071] Optionally, the attention level can be pre-set based on the type of abnormal behavior event, and can be divided into three levels of attention level (high attention level, low attention level, and no attention level). For example, this month, the focus is on abnormal behavior events such as running a red light, which can be set to high attention level, abnormal behavior events such as driving on the phone and not wearing a seat belt can be set to medium attention level, and abnormal behavior events such as speeding and crossing a solid line can be set to no attention level. Specifically, different attention levels can correspond to different scores, as shown in Table 2.

[0072] Table 2: Attention Scoring Rules

[0073] Attention Points High attention 4 Medium attention 2 No attention 0

[0074] 303: Determine whether the abnormal behavior event meets the requirements according to the severity level and the corresponding attention level corresponding to the abnormal behavior event.

[0075] Optionally, based on the scoring rule tables in Table 1 and Table 2 above, the score corresponding to the abnormal behavior event may be calculated according to the following formula:

[0076] C=a*b

[0077] Among them, C is the score corresponding to the abnormal behavior event, a is the score corresponding to the severity level of the abnormal behavior event, and b is the score corresponding to the attention level of the abnormal behavior event.

[0078] For example, the abnormal events identified in step 301 include event 1 with a very high severity level, event 2 with a high severity level, event 3 with a medium severity level, and event 4 with a low severity level. In step 302, event 1 is an abnormal behavior event with a high degree of concern, event 2 is an abnormal behavior event with a low degree of concern, and events 3 and 4 are abnormal behavior events with no concern. Based on the above formula, the scores corresponding to events 1, 2, 3, and 4 can be determined respectively, as shown in Table 3.

[0079] Table 3: Abnormal behavior event score calculation table

[0080]

[0081] If the calculated score corresponding to the abnormal behavior event is greater than the set threshold (eg, 10), it is determined that the abnormal behavior event (event 1) meets the requirements, and the process proceeds to step 304. If the abnormal behavior event does not meet the requirements, the process ends.

[0082] In some embodiments, if the scores corresponding to multiple abnormal behavior events are greater than a set threshold, for example, both event 1 and event 2 meet the requirements, the video frames containing event 1 and event 2 are identified separately to obtain the target object related to event 1 and the target object related to event 2.

[0083] In some embodiments, the abnormal behavior event with the highest score may also be determined as the abnormal behavior event that meets the requirements.

[0084] In other embodiments, the severity score and the attention score corresponding to the abnormal behavior event may be weighted and summed to determine whether the abnormal behavior event meets the requirements.

[0085] 304: Identify the video frames containing the abnormal behavior event in the first video sequence to obtain a target object related to the abnormal behavior event.

[0086] In an embodiment of the present application, based on the severity level and the degree of attention, it is determined whether the abnormal behavior event in the first video meets the requirements. Compared with identifying and determining the target object one by one when there are multiple events in the video sequence, it is possible to highlight the key points and quickly determine the target object, thereby improving efficiency.

[0087] 202: After the target object is determined, a second video sequence is generated according to the time when the abnormal behavior event of the target object occurs in the first video sequence.

[0088] In some embodiments, the second video sequence may include a video sequence when the target object has an abnormal behavior, and a video sequence of a set time period before the target object has the abnormal behavior.

[0089] In other embodiments, the second video sequence may include a video sequence when the target object exhibits abnormal behavior, and a video sequence of a set time after the target object exhibits the abnormal behavior.

[0090] In other embodiments, the second video sequence may include a video sequence when the target object exhibits abnormal behavior, and a video sequence of a set time length before and after the target object exhibits the abnormal behavior.

[0091] In an embodiment of the present application, not only the video sequence when the target object exhibits abnormal behavior is included, but also the video sequence before and / or after the abnormal behavior occurs within a set time period. Compared with the video sequence that only includes the video sequence when the abnormal behavior occurs, the focus can be on the trajectory of the target object, which facilitates the improvement of the details of the abnormal behavior event.

[0092] In some embodiments, the acquired second video sequence can be labeled (for example, case009), and the video sequence when the target object occurs abnormal behavior can also be labeled (for example, Z hours in the event, used to characterize the time period to which the result of the abnormal behavior event belongs), the video sequence before the target object occurs the abnormal behavior can be labeled (for example, X hours before the event, used to characterize the time period to which the cause of the abnormal behavior event belongs), and the video sequence after the target object occurs the abnormal behavior can be labeled (for example, Y hours after the event, used to characterize the time period to which the details of the abnormal behavior event belong), so as to distinguish the process of the occurrence of the abnormal behavior event.

[0093] 203: Cluster the video frames in the second video sequence to obtain at least one video subsequence associated with the target object.

[0094] Optionally, the second video sequence can be clustered by at least one algorithm such as k-means clustering algorithm, K-Nearest Neighbor (KNN), Expectation Maximization Algorithm (EMA), Density-Based Spatial Clustering of Applications with Noise (DBSCAN), etc., and the video subsequences related to the target object are retained, and other video subsequences not related to the target object can be deleted or ignored.

[0095] 204: After obtaining at least one video subsequence associated with the target object, determine the relevance of each video subsequence with the target object and the abnormal behavior event, and select a target video subsequence from the video subsequences according to the relevance of each video subsequence.

[0096] Taking any one of the video subsequences as an example, for the convenience of description, it is called the first video subsequence. Optionally, the correlation between each video subsequence and the target object and the abnormal behavior event can be determined in the following manner:

[0097] Method 1: Determine the correlation between the first video subsequence and the target object and the abnormal behavior event according to the time correlation, event main line correlation, and target object correlation of the first video subsequence.

[0098] Method 2: Determine the correlation between the first video subsequence and the target object and the abnormal behavior event based on any two of the time correlation, event main line correlation, and target object correlation of the first video subsequence.

[0099] Method three: Determine the correlation between the first video subsequence and the target object and the abnormal behavior event according to any one of the time correlation, event main line correlation, and target object correlation of the first video subsequence.

[0100] Optionally, the time correlation of the first video subsequence can be determined according to the interval between the time of the first video subsequence and the time when the abnormal behavior event occurs. If the interval between the time of the first video subsequence and the time when the abnormal behavior occurs is closer, the time correlation is higher, and if the interval between the time of the first video subsequence and the time when the abnormal behavior occurs is farther, the time correlation is lower. Optionally, the time correlation can be quantified by a score, for example, if the time period of the first video subsequence belongs to the video sequence range of "Z hours in the event" in the above label, it corresponds to 3 points; if the time period of the first video subsequence belongs to the video sequence range of "X hours before the event" in the above label, it corresponds to 2 points; if the time period of the first video subsequence belongs to the video sequence range of "Y hours after the event" in the above label, it corresponds to 1 point.

[0101] Optionally, the event main line relevance of the first video subsequence can be determined according to the event main line type to which the abnormal behavior event in the first video subsequence belongs, and the event main line type can be identified by an algorithm such as a target detection algorithm (e.g., YOLO v3). Optionally, the event main line type may include at least one of a cause, a result, and a detail, and the event main line relevance may be divided into scores according to the importance of the abnormal event itself (abnormal event chronicle rule), and generally, the result of an abnormal behavior event> the cause> the details, and the details may not be included according to actual conditions.

[0102] Optionally, the relevance of the target object of the first video subsequence is determined based on the frequency of the target object appearing in the first video subsequence. If the frequency of the target object appearing in the first video sequence is more, the target object relevance is higher, and vice versa. Optionally, the number of times the target object appears per unit time can be divided into three levels of high, medium and low, and corresponding to different scores. For example, if the target object appears more than 5 times per unit time, it is high, less than 3 times, it is low, and 3-5 times, it is medium.

[0103] Table 4 exemplarily shows the correlation scoring rule table of the above three different dimensions.

[0104] Table 4: Relevance scoring rules table

[0105]

[0106] Optionally, taking the first method as an example, the correlation between the first video subsequence and the target object and the abnormal behavior event satisfies the following formula:

[0107]

[0108] Among them, G represents the score corresponding to the correlation between the first video subsequence and the target object and the abnormal behavior event, d represents the score corresponding to the time correlation of the first video subsequence, e represents the score corresponding to the event main line correlation of the first video subsequence, and f represents the score corresponding to the target object correlation of the first video subsequence.

[0109] Through the above steps, the relevance of each video subsequence with the target object and the abnormal behavior event can be determined, and according to the relevance of each video subsequence, the target video subsequence is selected (as shown in Table 5), so that unnecessary video subsequences (such as video subsequences below the set score) can be deleted to shorten the video review time.

[0110] Table 5: Example of video subsequence score calculation table

[0111]

[0112] 205: Splicing the above target video subsequences to obtain a target video sequence associated with the target object and the abnormal behavior event.

[0113] Optionally, the target video subsequences may be spliced ​​in descending order of relevance. In some embodiments, if target sub-video sequences with the same score appear, they are spliced ​​according to the principle of Z hours in the event > hours after the event > Z hours before the event, thereby focusing on the results of the abnormal behavior event and improving the efficiency of processing the abnormal behavior event and the target object related to the event.

[0114] In an embodiment of the present application, video frames in a first video sequence are identified to obtain abnormal behavior events of a specified type in the first video sequence and target objects related to the abnormal behavior events, so as to highlight the key points and accurately identify the type of the abnormal behavior event and the related target objects; based on the time when the abnormal behavior event occurs in the target object in the first video sequence, a second video sequence is obtained, and the second video sequence is clustered to obtain at least one video subsequence associated with the target object, so as to focus on the trajectory of the target object and facilitate the improvement of the details of the abnormal behavior event; after determining the relevance of each video subsequence, and based on the relevance of each video subsequence, a target video subsequence is selected from the video subsequences associated with the target object, so as to delete unnecessary videos and shorten the video review time; the target video subsequences are spliced ​​to obtain a target video sequence associated with the target object and the abnormal behavior event, so as to focus on the result of the abnormal behavior event, thereby improving the efficiency of processing abnormal behavior events and related target objects.

[0115] based on Figure 2 The method flow shown, Figure 4 An exemplary video processing logic diagram is shown. As shown in the figure, after receiving an acquisition request including location A and time 9:00-10:00, a first video sequence is obtained, the video sequence is identified, and it is obtained that between 9:30 and 9:40, vehicle A ran a red light, resulting in a rear-end collision with vehicle B. According to this time period, a second video sequence from 7:30 to 11:40 is obtained, and the second sequence is clustered to obtain n video subsequences associated with vehicle A and / or B, and the correlation of the n video subsequences is calculated respectively to obtain 9 target video subsequences, which are spliced ​​in descending order of correlation to form a target video sequence.

[0116] Based on the same technical concept, the embodiment of the present application provides a structural diagram of a video processing device, such as Figure 5 As shown, the device includes a determination module 101b, an acquisition module 101c, a clustering module 502, a target video subsequence screening module 503, and a splicing module 504.

[0117] The determination module 101b is used to identify the video frames in the first video sequence, and obtain the abnormal behavior events of the specified type in the first video sequence and the target objects related to the abnormal behavior events.

[0118] The acquisition module 101c generates a second video sequence according to the time when the abnormal behavior event occurs to the target object in the first video sequence, wherein the second video sequence includes a video sequence when the target object occurs the abnormal behavior, and a video sequence of a set time before and / or after the target object occurs the abnormal behavior.

[0119] The clustering module 502 is configured to cluster the video frames in the second video sequence to obtain at least one video subsequence associated with the target object.

[0120] The target video subsequence screening module 503 is used to determine the correlation between the video subsequence and the target object and the abnormal behavior event, and select a target video subsequence from the video subsequences according to the correlation between each video subsequence.

[0121] The splicing module 504 is configured to splice the target video subsequences to obtain a target video sequence associated with the target object and the abnormal behavior event.

[0122] Optionally, the target video subsequence screening module 503 is specifically used for:

[0123] The correlation between the first video subsequence and the target object and the abnormal behavior event is determined according to at least one of the time correlation, event main line correlation, and target object correlation of the first video subsequence, and the first video subsequence is any one of the video subsequences.

[0124] Optionally, the determination module 101b is specifically configured to:

[0125] The video frames in the first video sequence are identified to obtain the abnormal behavior event and the severity level corresponding to the abnormal behavior event.

[0126] The correspondence between the preset event type and the attention degree is queried according to the type to which the abnormal behavior event belongs, and the attention degree corresponding to the abnormal behavior event is obtained.

[0127] According to the severity level and the corresponding attention level of the abnormal behavior event, it is determined whether the abnormal behavior event meets the requirements.

[0128] If the abnormal behavior event meets the requirements, the video frame containing the abnormal behavior event in the first video sequence is identified to obtain a target object related to the abnormal behavior event.

[0129] Optionally, the determination module 101b is specifically configured to:

[0130] The score corresponding to the abnormal behavior event is calculated according to the following formula:

[0131] C=a*b

[0132] Among them, C is the score corresponding to the abnormal behavior event, a is the score corresponding to the severity level of the abnormal behavior event, and b is the score corresponding to the attention level of the abnormal behavior event.

[0133] If the score corresponding to the abnormal behavior event is greater than a set threshold, it is determined that the abnormal behavior event meets the requirement.

[0134] Optionally, the splicing module 504 is specifically used for:

[0135] The target video subsequences are spliced ​​in descending order of relevance.

[0136] It should be noted here that the above-mentioned device provided in the embodiment of the present application can implement all the method steps in the above-mentioned video processing method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0137] Figure 6 The structural diagram of the communication device provided in the embodiment of the present application is exemplified.

[0138] As shown in the figure, the device may include: a processor 601 , a memory 602 , and a bus interface 603 .

[0139] The processor 601 is responsible for managing the bus architecture and general processing, and the memory 602 can store data used by the processor 601 when performing operations.

[0140] The bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 601 and memory represented by memory 602. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The processor 601 is responsible for managing the bus architecture and general processing, and the memory 602 may store data used by the processor 601 when performing operations.

[0141] The process disclosed in the embodiment of the present application can be applied to the processor 601, or implemented by the processor 601. In the implementation process, each step of the signal processing process can be completed by the hardware integrated logic circuit or software instructions in the processor 601. The processor 601 can be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method applied in conjunction with the embodiment of the present application can be directly embodied as a hardware processor to be executed, or the hardware and software modules in the processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 602, and the processor 601 reads the information in the memory 602 and completes the steps of the information processing process in conjunction with its hardware.

[0142] Specifically, the processor 601 is used to read the computer instructions in the memory 602 and execute a video processing method in the embodiment of the present application.

[0143] It should be noted here that the above-mentioned communication device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned method embodiment, and can achieve the same technical effect. The parts and beneficial effects that are the same as those in the method embodiment will not be described in detail here.

[0144] An embodiment of the present application further provides a computer-readable storage medium, which stores computer-executable instructions. The computer-executable instructions are used to enable a computer to execute a video processing method in the above embodiment.

[0145] An embodiment of the present application further provides a computer program product, which, when called by a computer, enables the computer to execute a video processing method in the above embodiment.

[0146] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0147] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0148] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0149] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0150] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A video processing method, characterized in that: include: Identify video frames in a first video sequence to obtain abnormal behavior events of a specified type in the first video sequence and target objects related to the abnormal behavior events; Generate a second video sequence according to the time when the target object in the first video sequence has the abnormal behavior event, wherein the second video sequence includes a video sequence when the target object has the abnormal behavior, and a video sequence of a set time before and / or after the target object has the abnormal behavior; Clustering the video frames in the second video sequence to obtain at least one video subsequence associated with the target object; Determining the relevance of the video subsequences with the target object and the abnormal behavior event, and selecting a target video subsequence from the video subsequences according to the relevance of each video subsequence; The target video subsequences are spliced ​​to obtain a target video sequence associated with the target object and the abnormal behavior event.

2. The method according to claim 1, characterized in that The determining of the relevance of each video subsequence in the at least one video subsequence associated with the target object to the target object and the abnormal behavior event comprises: The correlation between the first video subsequence and the target object and the abnormal behavior event is determined according to at least one of the time correlation, event main line correlation, and target object correlation of the first video subsequence, and the first video subsequence is any one of the video subsequences.

3. The method according to claim 2, characterized in that The correlation between the first video subsequence, the target object and the abnormal behavior event satisfies the following formula: Among them, G represents the score corresponding to the correlation between the first video subsequence and the target object and the abnormal behavior event, d represents the score corresponding to the time correlation of the first video subsequence, e represents the score corresponding to the event main line correlation of the first video subsequence, and f represents the score corresponding to the target object correlation of the first video subsequence.

4. The method according to claim 1, characterized in that The step of identifying video frames in the first video sequence to obtain an abnormal behavior event of a specified type in the first video sequence and a target object related to the abnormal behavior event includes: Identify the video frames in the first video sequence to obtain the abnormal behavior event and the severity level corresponding to the abnormal behavior event; According to the type of the abnormal behavior event, query the correspondence between the preset event type and the attention degree to obtain the attention degree corresponding to the abnormal behavior event; Determine whether the abnormal behavior event meets the requirements according to the severity level and the corresponding attention level of the abnormal behavior event; If the abnormal behavior event meets the requirements, the video frame containing the abnormal behavior event in the first video sequence is identified to obtain a target object related to the abnormal behavior event.

5. The method according to claim 4, characterized in that The determining, according to the severity level and the corresponding attention level of the abnormal behavior event, whether the abnormal behavior event meets the requirements includes: The score corresponding to the abnormal behavior event is calculated according to the following formula: C=a*b Wherein, C is the score corresponding to the abnormal behavior event, a is the score corresponding to the severity level of the abnormal behavior event, and b is the score corresponding to the attention level of the abnormal behavior event; If the score corresponding to the abnormal behavior event is greater than a set threshold, it is determined that the abnormal behavior event meets the requirement.

6. The method according to any one of claims 1 to 5, characterized in that: The step of splicing the target video subsequences comprises: The target video subsequences are spliced ​​in descending order of relevance.

7. A video processing device, characterized in that: include: A determination module, configured to identify video frames in a first video sequence, and obtain an abnormal behavior event of a specified type in the first video sequence and a target object related to the abnormal behavior event; an acquisition module, generating a second video sequence according to the time when the target object in the first video sequence has the abnormal behavior event, wherein the second video sequence includes a video sequence when the target object has the abnormal behavior, and a video sequence of a set time before and / or after the target object has the abnormal behavior; a clustering module, configured to cluster the video frames in the second video sequence to obtain at least one video subsequence associated with the target object; A target video subsequence screening module is used to determine the relevance of the video subsequence with the target object and the abnormal behavior event, and select a target video subsequence from the video subsequences according to the relevance of each video subsequence; The splicing module is used to splice the target video subsequences to obtain a target video sequence associated with the target object and the abnormal behavior event.

8. The device according to claim 7, characterized in that The target video subsequence screening module is specifically used for: The correlation between the first video subsequence and the target object and the abnormal behavior event is determined according to at least one of the time correlation, event main line correlation, and target object correlation of the first video subsequence, and the first video subsequence is any one of the video subsequences.

9. The device according to claim 8, characterized in that The correlation between the first video subsequence, the target object and the abnormal behavior event satisfies the following formula: Among them, G represents the score corresponding to the correlation between the first video subsequence and the target object and the abnormal behavior event, d represents the score corresponding to the time correlation of the first video subsequence, e represents the score corresponding to the event main line correlation of the first video subsequence, and f represents the score corresponding to the target object correlation of the first video subsequence.

10. The device according to claim 7, characterized in that The determining module is specifically used for: Identify the video frames in the first video sequence to obtain the abnormal behavior event and the severity level corresponding to the abnormal behavior event; According to the type of the abnormal behavior event, query the correspondence between the preset event type and the attention degree to obtain the attention degree corresponding to the abnormal behavior event; Determine whether the abnormal behavior event meets the requirements according to the severity level and the corresponding attention level of the abnormal behavior event; If the abnormal behavior event meets the requirements, the video frame containing the abnormal behavior event in the first video sequence is identified to obtain a target object related to the abnormal behavior event.

11. The device according to claim 10, characterized in that The determining module is specifically used for: The score corresponding to the abnormal behavior event is calculated according to the following formula: C=a*b Wherein, C is the score corresponding to the abnormal behavior event, a is the score corresponding to the severity level of the abnormal behavior event, and b is the score corresponding to the attention level of the abnormal behavior event; If the score corresponding to the abnormal behavior event is greater than a set threshold, it is determined that the abnormal behavior event meets the requirement.

12. The device according to any one of claims 7 to 11, characterized in that: The splicing module is specifically used for: The target video subsequences are spliced ​​in descending order of relevance.

13. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

14. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computer device, and when the program is run on the computer device, the computer device executes the method according to any one of claims 1 to 6.

15. A computer program product, characterized in that The computer program product comprises a computer program stored on a computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer is caused to execute the method according to any one of claims 1 to 6.

Images