Identification method, device, electronic device and computer readable storage medium

Abstract

The embodiments of the present disclosure disclose a recognition method, device, electronic device, and computer-readable storage medium. The method includes: performing target detection on the current video frame to determine the bounding box area of ​​at least one target object; determining the focus area of ​​the current video frame according to the bounding box area; obtaining the historical focus area of ​​the historical video frame sequence, and determining the category of the focus area according to the intersection-and-union ratio between the focus area and the historical focus area; the historical video frame sequence is a video frame sequence that is before the current video frame and has been detected; determining the trajectory sequence corresponding to each type of focus area from the historical video frame sequence and the current video frame; identifying the trajectory sequence corresponding to each type of focus area, and obtaining the corresponding recognition result. Through the present disclosure, the recognition accuracy and recognition efficiency can be improved.

Description

Technical Field

[0001] The present disclosure relates to computer vision technology, and more particularly to a recognition method, device, electronic device, and computer-readable storage medium. Background Art

[0002] Anomaly detection and recognition in videos is an important problem in the field of computer vision. It has a wide range of applications in the field of intelligent recognition, such as detecting and identifying illegal behaviors, traffic accidents, and some uncommon events. Thousands of cameras are deployed around the world. However, most cameras only record the dynamics of each moment and do not have the ability to automatically identify (often special personnel are required to be responsible for manual inspection). Due to the huge number of videos, it is obviously unrealistic to filter the content in the video by manpower alone. We need to use computer vision and deep learning technology to automatically detect and identify abnormal events occurring in the video. However, it is extremely difficult to identify abnormal events in the video collected by the camera. Possible challenges include scarcity of labeled data due to low-probability events, large variance between / within classes, subjective definition differences of abnormal events, low resolution of videos, etc.

[0003] In the related art, a trained network model is usually used to identify abnormal events in the video captured by the camera, but the efficiency and accuracy of the recognition are low, and abnormal events in the video cannot be discovered in a timely and accurate manner. Summary of the invention

[0004] The embodiments of the present disclosure provide an identification method, an apparatus, an electronic device, and a computer-readable storage medium, which can improve identification accuracy and efficiency.

[0005] The technical solution of the embodiment of the present disclosure is implemented as follows:

[0006] The present disclosure provides a recognition method, including: performing target detection on a current video frame to determine a bounding box area of ​​at least one target object; determining a focus area of ​​the current video frame according to the bounding box area; obtaining a historical focus area of ​​a historical video frame sequence, and determining a category of the focus area according to an intersection-and-union ratio between the focus area and the historical focus area; the historical video frame sequence is a video frame sequence that is before the current video frame and has been detected; determining a trajectory sequence corresponding to each category of focus area from the historical video frame sequence and the current video frame; identifying the trajectory sequence corresponding to each category of focus area to obtain a corresponding recognition result.

[0007] The embodiment of the present disclosure provides a recognition device, including: a detection unit, used to perform target detection on a current video frame, and determine a bounding box area of ​​at least one target object; a determination unit, used to determine a focus area of ​​the current video frame according to the bounding box area; obtain historical focus areas of a historical video frame sequence, and determine the category of the focus area according to an intersection-and-union ratio between the focus area and the historical focus area; the historical video frame sequence is a video frame sequence that is before the current video frame and has been detected; a segmentation unit, used to determine a trajectory sequence corresponding to each type of focus area from the historical video frame sequence and the current video frame; and an recognition unit, used to identify the trajectory sequence corresponding to each type of focus area to obtain a corresponding recognition result.

[0008] In some embodiments of the present disclosure, the determination unit is further used to determine whether there is a matching focus area in the historical focus area based on the intersection-over-union ratio between the focus area and the historical focus area; and determine the category of the focus area based on whether there is the matching focus area in the historical focus area.

[0009] In some embodiments of the present disclosure, the determination unit, when the matching area of ​​interest exists in the historical area of ​​interest, uses the area category of the matching area of ​​interest as the area category of the area of ​​interest; and / or, when the matching area of ​​interest does not exist in the historical area of ​​interest, sets a new area category for the area of ​​interest, and the new area category is different from the area category of the historical area of ​​interest.

[0010] In some embodiments of the present disclosure, the segmentation unit is further used to extract the focus area and the historical focus area from each video frame containing each type of focus area in the historical video frame sequence and the current video frame to obtain a focus area image; based on the focus area image, obtain a trajectory sequence corresponding to each type of focus area; wherein, in the case where any focus area has at least two area categories, the trajectory sequence corresponding to each type of focus area in the at least two area categories includes a focus area image corresponding to any focus area.

[0011] In some embodiments of the present disclosure, each bounding box area corresponds to a target identifier; the intersection-and-union ratio includes: a target identifier intersection-and-union ratio; the determination unit is also used to determine the target identifier intersection-and-union ratio between each focus area and each historical focus area; when at least one target identifier intersection-and-union ratio is greater than or equal to a preset intersection-and-union ratio threshold, it is determined that there is at least one matching focus area in the historical focus area; when all target identifier intersection-and-union ratios are less than the preset intersection-and-union ratio threshold, it is determined that there is no matching focus area in the historical focus area.

[0012] In some embodiments of the present disclosure, the determination unit is also used to determine, for any focus area and any historical focus area, a first target identifier corresponding to each bounding box area contained in any focus area to obtain at least one first target identifier; and determine a second target identifier corresponding to each bounding box area contained in any historical focus area to obtain at least one second target identifier; determine a first number of identical target identifiers and a second number of different target identifiers in the at least one first target identifier and the at least one second target identifier; determine the sum of the first number and the second number, and use the ratio of the first number to the sum as the target identifier intersection-union ratio between any focus area and any historical focus area.

[0013] In some embodiments of the present disclosure, the determination unit is further used to select a video frame from the historical video frame sequence that contains at least one focus area and whose detection order is closest to the current video frame as a historical video frame; and determine the category of the focus area based on whether the matching focus area exists in the historical focus areas of the historical video frame.

[0014] In some embodiments of the present disclosure, the target object includes a real person; the recognition unit is further used to use a classification network to classify and recognize the trajectory sequence corresponding to each type of focus area, so as to obtain the person behavior corresponding to the trajectory sequence corresponding to each type of focus area.

[0015] An embodiment of the present disclosure provides an electronic device, comprising: a memory for storing an executable computer program; and a processor for implementing the above-mentioned identification method in combination with the display screen when executing the executable computer program stored in the memory.

[0016] The embodiment of the present disclosure provides a computer-readable storage medium storing a computer program for causing a processor to execute the above-mentioned identification method.

[0017] The embodiments of the present disclosure have the following beneficial effects:

[0018] Since, for each video frame containing a target object, the focus area of ​​the current video frame is determined according to the bounding box area of ​​the target object, and the category of the focus area is determined according to the intersection-and-union ratio between the historical focus area and the focus area in the video frame sequence that has been detected before the current video frame, the focus area of ​​each category is obtained. Therefore, the focus area of ​​each category is an important area in the corresponding video frame; and, for the obtained video frame sequence composed of the current video frame and the historical video frame sequence, according to the focus area of ​​each category contained in the video frames of the video frame sequence, the trajectory sequence of the focus area of ​​each category corresponding to the video frame sequence is obtained, so the trajectory sequence corresponding to the focus area of ​​each category is composed of important areas belonging to the same area in the video frame sequence; therefore, when the trajectory sequence of the focus area of ​​each category is identified, compared with identifying the entire image of each video frame in the video frame sequence, the interference of background factors in the image on the identification is reduced, and the calculation amount and recognition range during identification are reduced, thereby improving the recognition efficiency and improving the accuracy of the obtained recognition result.

[0019] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0021] Figure 1 An optional flowchart of the identification method provided in the embodiment of the present disclosure;

[0022] Figure 2 A schematic diagram of the relationship between an enlarged area and a focus area for a bounding box area provided exemplarily in an embodiment of the present disclosure;

[0023] Figure 3A and Figure 3B They are schematic diagrams of trajectory sequences corresponding to two types of regions of interest provided in the embodiments of the present disclosure;

[0024] Figure 4 An optional flowchart of the identification method provided in the embodiment of the present disclosure;

[0025] Figure 5 An optional flowchart of the identification method provided in the embodiment of the present disclosure;

[0026] Figure 6 An optional flowchart of the identification method provided in the embodiment of the present disclosure;

[0027] Figure 7 An optional flowchart of the identification method provided in the embodiment of the present disclosure;

[0028] Figure 8 An optional flowchart of the identification method provided in the embodiment of the present disclosure;

[0029] Fig. 9 A schematic diagram of an exemplary focus region image of a focus region provided in an embodiment of the present disclosure;

[0030] Fig.10 A schematic diagram of an area corresponding to the minimum coordinate union of two area coordinates provided by an exemplary embodiment of the present disclosure in a video frame;

[0031] Fig.11 A schematic diagram of the structure of an identification device provided in an embodiment of the present disclosure;

[0032] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] In order to make the objectives, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present disclosure. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0034] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0035] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs. The terms used herein are only for the purpose of describing the embodiments of the present disclosure and are not intended to limit the present disclosure.

[0037] Anomaly detection and recognition in videos is an important problem in the field of computer vision. It has a wide range of applications in the field of intelligent recognition, such as detecting and identifying illegal behaviors, traffic accidents, and some uncommon events. Thousands of cameras are deployed around the world. However, most cameras only record the dynamics of each moment and do not have the ability to automatically identify (often special personnel are required to be responsible for manual inspection). Due to the huge number of videos, it is obviously unrealistic to filter the content in the video by manpower alone. We need to use computer vision and deep learning technology to automatically detect and identify abnormal events occurring in the video. However, it is extremely difficult to identify abnormal events in the video collected by the camera. Possible challenges include scarcity of labeled data due to low-probability events, large variance between / within classes, subjective definition differences of abnormal events, low resolution of videos, etc.

[0038] In the related art, when a trained network model is used to identify abnormal events in videos captured by a camera, the entire image data of each frame of video in the input video sequence is usually enhanced or otherwise preprocessed before prediction. This leads to more background factors, inaccurate classification, and a large amount of calculation, resulting in low recognition efficiency and accuracy, and inability to detect abnormal events in the video in a timely and accurate manner.

[0039] The embodiments of the present disclosure provide a recognition method that can improve the efficiency and accuracy of recognition. The recognition method provided by the embodiments of the present disclosure is applied to electronic devices. The exemplary applications of the electronic devices provided by the embodiments of the present disclosure are described below. The electronic devices provided by the embodiments of the present disclosure can be implemented as various types of user terminals (hereinafter referred to as terminals) such as AR glasses, laptops, tablet computers, desktop computers, set-top boxes, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), and can also be implemented as servers.

[0040] Figure 1 is an optional flow chart of the identification method provided by the embodiment of the present disclosure, which will be combined with Figure 1 The steps shown are explained.

[0041] S101: Perform target detection on a current video frame to determine a bounding box area of ​​at least one target object.

[0042] In the disclosed embodiment, the electronic device may use a pre-trained target detection model to perform target detection on the current video frame, determine whether there is a target object in the current video frame, and when there is at least one target object, determine the bounding box of each target object. Exemplarily, the target detection model may be a Faster R-CNN (Fast Region-based Convolutional Network, Faster R-CNN) model, or the target detection model may also be a Single Shot MultiBox Detector (SSD) or other target detection model, which is not limited in the disclosed embodiment.

[0043] In some embodiments, the current video frame is a video frame being detected in a sequence of video frames to be detected; when the electronic device is equipped with a camera, it can capture the sequence of video frames to be detected in real time through its own camera, and perform target detection on each video frame in the sequence of video frames to be detected in real time; alternatively, the electronic device can also obtain the sequence of video frames to be detected from other devices in real time, and perform target detection on each video frame in the sequence of video frames to be detected acquired in real time.

[0044] In the embodiments of the present disclosure, the video frame may be a video image of any scene, for example, a video image of a street or a video image of a shopping mall, which is not limited in the embodiments of the present disclosure. In the embodiments of the present disclosure, the target object may be any object, for example, a cat, a dog, a car or a real person, which is not limited in the embodiments of the present disclosure.

[0045] It should be noted that the current video frame may be a video frame containing at least one target object or a video frame not containing a target object. Here, the case where the current video frame contains at least one target object is discussed, and the case where the current video frame does not contain a target object will be explained in other parts of the present disclosure.

[0046] S102: Determine a focus area of ​​the current video frame according to the bounding box area.

[0047] In the embodiment of the present disclosure, when the electronic device determines the bounding box area in the current video frame, it can determine the focus area in the current video frame according to all the bounding box areas in the current video frame.

[0048] In some embodiments, when there is at least one target object in the current video frame, for each target object, the electronic device can obtain an enlarged area corresponding to the target object by enlarging the area of ​​the bounding box area corresponding to the target object, thereby obtaining at least one enlarged area corresponding to at least one target object, and can determine one or more enlarged areas from the at least one enlarged area as the key area of ​​the current video frame, and, for each key area, the electronic device can determine the focus area corresponding to the key area based on the bounding box area corresponding to the key area, thereby obtaining at least one focus area of ​​the current video frame; the focus area corresponding to each key area can be the same as the key area, or the focus area corresponding to each key area can also include the key area.

[0049] Here, the enlarged area of ​​each target object may be an area obtained by enlarging the area of ​​the bounding box area of ​​the target object by a first preset ratio; the focus area corresponding to each key area is an area obtained by enlarging the area of ​​the bounding box area of ​​the target object corresponding to the key area by a second preset ratio. The first preset ratio and the second preset ratio may be set according to actual needs, and the embodiments of the present disclosure are not limited thereto. For example, the first preset ratio may be 1.5, and the second preset ratio may be 2.

[0050] For example, Figure 2 It shows an enlarged area W obtained by enlarging the area of ​​a bounding box area Q of a target object by 1.5 times, and a focus area G corresponding to the enlarged area W obtained by enlarging the area of ​​the bounding box area Q by 2 times when the enlarged area W is a key area.

[0051] In some embodiments, when there is only one target object in the current video frame, the electronic device can directly use the enlarged area of ​​the target object as the focus area of ​​the current video frame after obtaining the enlarged area corresponding to the target object by enlarging the area of ​​the bounding box area corresponding to the target object.

[0052] In some embodiments, when there are at least two target objects in the current video frame, the electronic device may, after obtaining the corresponding at least two enlarged areas, select at least one enlarged area as a key area according to the number of target objects contained in each enlarged area.

[0053] In some embodiments, when there are at least two target objects in the current video frame, the electronic device can also use the following method to determine the key area in the current video frame: for each expanded area and each remaining expanded area, calculate the area intersection-union ratio between the expanded area and the remaining expanded area, and when the area intersection-union ratio is greater than or equal to a preset area intersection-union ratio threshold, determine that the expanded area matches the remaining expanded area; use the number of remaining expanded areas that match the expanded area as the number of matches between the expanded area and at least one remaining expanded area; use the top N expanded areas with the largest number of matches among at least two expanded areas as the key areas of the current video frame; wherein, at least one remaining expanded area is an expanded area other than the expanded area among the at least two expanded areas corresponding to at least two target objects; and N is an integer greater than or equal to 1. For example, for the current video frame, when the electronic device obtains three enlarged areas (enlarged area 1, enlarged area 2, and enlarged area 3), for enlarged area 1, the electronic device can calculate the number of matches between enlarged area 1 and enlarged area 2 and enlarged area 3. Specifically, for enlarged area 1, the electronic device can calculate the area intersection and union ratio between enlarged area 1 and enlarged area 2, and calculate the area intersection and union ratio between enlarged area 1 and enlarged area 3, and when the area intersection and union ratio between enlarged area 1 and enlarged area 2 is less than a preset area intersection and union ratio threshold, it is determined that enlarged area 1 does not match enlarged area 2, and when the area intersection and union ratio between enlarged area 1 and enlarged area 3 is greater than or equal to the preset area intersection and union ratio threshold, it is determined that enlarged area 1 matches enlarged area 3; thus, the number of matches for enlarged area 1 can be obtained to be 1. As for enlarged area 2 and enlarged area 3, the principle is the same as above, and the embodiments of the present disclosure are not described in detail here.

[0054] S103, obtaining a historical focus region of a historical video frame sequence, and determining a category of the focus region according to an intersection-over-union ratio between the focus region and the historical focus region; the historical video frame sequence is a video frame sequence that is before the current video frame and has been detected.

[0055] In the disclosed embodiment, for the current video frame, the electronic device determines that, if there is a detected video frame sequence before the current video frame, it can obtain the detected video frame sequence before the current video frame, and obtain the historical focus area in the video frame sequence, and calculate the intersection-and-union ratio between the obtained historical focus area and the focus area in the current video frame, and determine the category of the focus area in the current video frame according to the calculated intersection-and-union ratio.

[0056] In some embodiments, when the electronic device determines that there is no video frame sequence that has been detected before the current video frame, that is, the current video frame is an initial video frame, the electronic device can determine that the historical focus area of ​​the acquired historical video frame sequence is empty, and then all focus areas in the current video frame can be determined as focus areas of different regional categories for use when determining the regional category of the focus area in the next current video frame. It can be understood that focus areas with the same regional category belong to the same area; and areas with different regional categories belong to different areas.

[0057] In some embodiments, the electronic device can obtain any one video frame in a historical video frame sequence, and use the focus area in the any one video frame as the historical focus area to determine the area category of each focus area in the current video frame. It can also obtain any multiple video frames in a historical video frame sequence, and use the focus areas in the any multiple video frames as the historical focus areas to determine the area category of each focus area in the current video frame. The embodiments of the present disclosure are not limited to this.

[0058] It should be noted that the historical attention region is the attention region in the historical video frame. Here, the description of the historical attention region is used to distinguish it from the attention region in the current video frame.

[0059] In some embodiments, each historical focus area has a region identification (patch Identity document, patch ID), which is used to characterize the region category to which the historical focus area belongs; based on this, the above S103 can also be implemented in the following way: S1031, obtain the historical focus area of ​​the historical video frame sequence, and determine the region identification of the focus area based on the intersection and union ratio between the focus area and the historical focus area.

[0060] During the detection process, the electronic device sets a region identifier for each region of interest, and uses the region identifier to characterize the region category of the region of interest, wherein the regions of interest with the same region identifier have the same region category, indicating that they belong to the same region; and the regions of interest with different region identifiers have different region categories, indicating that they belong to different regions. The electronic device can determine the region identifier of each region of interest in the current video frame based on the historical regions of interest obtained from the historical video frame sequence, and based on the intersection-and-union ratio between the region of interest in the current video frame and the historical regions of interest, obtain the region category of each region of interest in the current video frame, and thereby obtain the correlation between each region of interest in the current video frame and the historical regions of interest, so as to obtain the corresponding trajectory sequence later.

[0061] S104: Determine a trajectory sequence corresponding to each type of focus area from the historical video frame sequence and the current video frame.

[0062] In the disclosed embodiment, after detecting a current video frame, the electronic device can determine all the focus area images corresponding to each category of the focus area from the video frame sequence composed of the historical video frame sequence and the current video frame, and use the sequence composed of all the corresponding focus area images as the trajectory sequence corresponding to the focus area of ​​the category. The trajectory sequence corresponding to each category of the focus area is composed of the image parts corresponding to all the focus areas of the category in the video frame. For example, when the trajectory sequence corresponding to the focus area with area category A is composed of the focus area image m1 and the focus area image m2, and the focus area image m1 belongs to the video frame M1, and the focus area image m2 belongs to the video frame M2, the focus area image m1 is the image part corresponding to the focus area with area category A in the video frame M1, and the focus area image m2 is the image part corresponding to the focus area with area category A in the video frame M2.

[0063] In some embodiments, the electronic device can obtain a video frame sequence composed of a historical video frame sequence and a current video frame when it is determined that a preset condition is met, and determine the trajectory sequence corresponding to each category of the focus area from the obtained video frame sequence. Exemplarily, the electronic device can determine that the preset condition is met when it is determined that the detected time has reached a preset time, and obtain the current video frame according to a current video frame corresponding to the preset time, and obtain the historical video frame sequence corresponding to the current video frame, so as to obtain the trajectory sequence of each category of the focus area. The preset time can be set according to actual needs, for example, it can be 6 seconds, then the electronic device can obtain the current video frame corresponding to the 6th second, and obtain the historical video frame sequence corresponding to the current video frame (that is, obtain all video frames detected within these 6 seconds), and obtain the trajectory sequence corresponding to each category of the focus area according to the current video frame and the historical video frame sequence corresponding to the current video frame; in this way, the trajectory sequence corresponding to each category of the focus area can be obtained regularly, which is conducive to the timely identification of the trajectory sequence.

[0064] For example, Figure 3A and Figure 3B are trajectory sequences corresponding to the two types of regions of interest provided in the embodiments of the present disclosure. Figure 3A and 3B As shown, when the target object is a real person, the region category is a( Figure 3A The trajectory sequence corresponding to the focus area (not shown) is G1, and the area category is b ( Figure 3BThe trajectory sequence corresponding to the focus area (not shown) is G2; and each trajectory sequence contains important image parts belonging to the same area, so the electronic device can classify and identify the obtained trajectory sequences to achieve fast and accurate identification of each trajectory sequence.

[0065] S105 , identifying the trajectory sequence corresponding to each type of focus area, and obtaining a corresponding recognition result.

[0066] In the disclosed embodiment, after obtaining the trajectory sequence corresponding to each category of the focus area, the electronic device may identify the trajectory sequence corresponding to each category of the focus area, thereby obtaining an identification result.

[0067] In some embodiments, the electronic device may use a classification network (classification model) to classify the trajectory sequence corresponding to each category of the focus area, and use the classification result as the final recognition result, wherein the classification network may be a residual network (ResNet) or other networks, which is not limited in the embodiments of the present disclosure.

[0068] In some embodiments, the target object includes a real person; the above S105 can be implemented in the following manner: using a classification network to classify and identify the trajectory sequence corresponding to each type of focus area, and obtain the person behavior corresponding to the trajectory sequence.

[0069] For each type of track sequence corresponding to the focus area, the electronic device can use a classification network to classify the track sequence corresponding to the focus area of ​​the type, and obtain the character behavior corresponding to the track sequence corresponding to the focus area of ​​the type, such as fighting behavior or making trouble. Figure 3A The recognition result of the trajectory sequence G1 in can be: normal walking; Figure 3A The recognition result of the trajectory sequence G2 in can be: fighting.

[0070] In some embodiments, Figure 4 is an optional flow chart of the identification method provided by the embodiment of the present disclosure, such as Figure 4 As shown, the above S103 can also be implemented by S1032-S1033, combining Figure 4 The steps in are described below.

[0071] S1032: Obtain the historical attention region of the historical video frame sequence, and determine whether there is a matching attention region in the historical attention region according to the intersection-over-union ratio between the attention region and the historical attention region.

[0072] S1033: Determine the category of the attention area according to whether there is a matching attention area in the historical attention areas.

[0073] In an embodiment of the present disclosure, for a current video frame, when the electronic device determines that there is a detected video frame sequence before the current video frame, the detected video frame sequence before the current video frame can be used as a historical video sequence, and based on whether there is a matching focus area in at least one historical focus area in the historical video frame sequence, the area category of each focus area in the current video frame is determined, thereby obtaining the area category of each focus area in the current video frame.

[0074] In the disclosed embodiment, each matching focus region is a historical focus region in a historical video frame sequence, and for any focus region, the intersection-and-union ratio between each matching focus region and the any focus region satisfies a preset intersection-and-union ratio threshold. It can be understood that the matching focus region is targeted, and for any focus region, each matching focus region for the focus region is a historical focus region whose intersection-and-union ratio with the focus region satisfies a preset intersection-and-union ratio threshold. For example, for a focus region m in the current video frame, each matching focus region for the focus region m is a historical focus region whose intersection-and-union ratio with the focus region m satisfies a preset intersection-and-union ratio threshold.

[0075] Here, for any focus area in the current video frame, when the electronic device determines that there is a matching focus area corresponding to the focus area in the historical video frame sequence, at least one region category will be determined for the focus area, and when the electronic device determines that there is no matching focus area corresponding to the focus area in the historical video frame sequence, at least one other region category will be determined for the focus area. For example, for the above-mentioned focus area m, when the electronic device determines that there is a matching focus area corresponding to the focus area m in the historical video frame sequence, the region category of the focus area m will be determined as region category A, and when the electronic device determines that there is no matching focus area corresponding to the focus area m in the historical video frame sequence, the region category of the focus area m will be determined as region category B, and region categories A and B belong to different region categories.

[0076] In some embodiments, Figure 5 is an optional flow chart of the identification method provided by the embodiment of the present disclosure, such as Figure 5 As shown, the above S1032 can be implemented by S201-S203, combining Figure 5 The steps in are described below.

[0077] S201, obtaining historical focus areas of a historical video frame sequence, and determining a target identifier intersection-over-union ratio between each focus area and each historical focus area; each bounding box area corresponds to a target identifier; and the intersection-over-union ratio includes a target identifier intersection-over-union ratio.

[0078] In the disclosed embodiment, a bounding box area of ​​each target object corresponds to a target identifier, and the bounding box areas of different target objects correspond to different target identifiers, while the bounding box areas of the same target objects correspond to the same target identifiers. For example, when the target object is a real person, the target identifier corresponding to the bounding box area of ​​person 1 is ID1, and the target identifier corresponding to the bounding box area of ​​person 2 is ID2.

[0079] In the disclosed embodiment, each focus area contains a target object, and the electronic device can determine the corresponding target identifier intersection-and-union ratio based on the intersection-and-union ratio between the target identifier of the target object contained in each focus area in the current video frame and the target identifier of the target object contained in each historical focus area. The target identifier intersection-and-union ratio between a focus area and a historical focus area is: the ratio between the number of identical target identifiers contained in the focus area and the historical focus area and the total number of target identifiers contained in the focus area and the historical focus area.

[0080] In some embodiments, Figure 6 As shown, for any concerned area and any historical concerned area, the above S201 can be implemented through S2011-S2013; Figure 6 The steps in are described below.

[0081] S2011, determining a first target identifier corresponding to each bounding box area included in any focus area to obtain at least one first target identifier; and determining a second target identifier corresponding to each bounding box area included in any historical focus area to obtain at least one second target identifier.

[0082] For any focus area in the current video frame and any historical focus area in the historical video frame sequence, the electronic device can determine all first target identifiers corresponding to all target objects contained in the focus area, thereby obtaining at least one first target identifier; and determine all second target identifiers corresponding to all target objects contained in the historical focus area, thereby obtaining at least one second target identifier. For example, the electronic device can determine that a focus area H in the current video frame contains target identifier ID1 and target identifier ID2, and a historical focus area L in the historical video frame sequence contains target identifier ID1 and target identifier ID3.

[0083] S2012: Determine a first number of identical target identifiers and a second number of different target identifiers among at least one first target identifier and at least one second target identifier.

[0084] When the electronic device determines all the first target identifiers in the focus area and all the second target identifiers in the historical focus area respectively, it can determine the first number of identical target identifiers and the second number of different target identifiers among all the first target identifiers and all the second target identifiers; for example, with respect to the above, the electronic device determines that the focus area H contains target identifier ID1 and target identifier ID2, and the historical focus area L contains target identifier ID1 and target identifier ID3, then the electronic device can determine that in the focus area H and the historical focus area L, there is 1 identical target identifier (i.e., ID1) and 2 different target identifiers (i.e., ID2 and ID3).

[0085] S2013: Determine the sum of the first number and the second number, and use the ratio of the first number to the sum as the target identifier intersection-and-union ratio between the any concerned region and the any historical concerned region.

[0086] After determining the first number and the second number, the electronic device can sum the first number and the second number to obtain the sum of the numbers, and calculate the ratio between the first number and the sum of the numbers, thereby obtaining the target identifier intersection-and-union ratio between the area of ​​interest and the historical area of ​​interest. For example, when the electronic device determines that there is one identical target identifier (i.e., ID1) and two different target identifiers (i.e., ID2 and ID3) in the area of ​​interest H and the historical area of ​​interest L, the sum of the first number 1 and the second number 2 is 3, and thus, it can be determined that the target identifier intersection-and-union ratio between the area of ​​interest H and the historical area of ​​interest L is 1 / 3.

[0087] In the disclosed embodiment, when the target object is a real person, the above method can be used to calculate the overlap of the IDs of the real person, so that the trajectory sequence corresponding to the area containing multiple people can be obtained, which improves the effective receptive field of the event while expanding the search range, and improves the recall rate of the pre-processing of the video frames that need to be detected before identifying the video frames.

[0088] The following continues Figure 5 S202-S203 in the figure are explained as follows: S202: when the intersection-over-union ratio of at least one target identifier is greater than or equal to a preset intersection-over-union ratio threshold, it is determined that there is at least one matching focus area in the historical focus areas.

[0089] In the disclosed embodiment, for each focus area in the current video frame, when the electronic device determines that in the historical video frame sequence, there are one or more historical focus areas, and the target identification intersection-and-union ratio between the focus area and the electronic device is greater than or equal to a preset intersection-and-union ratio threshold, the electronic device can use the one or more historical focus areas as a matching focus area or multiple matching focus areas, and determine that there is at least one matching focus area for the focus area among the historical focus areas in the historical video frame sequence.

[0090] In the embodiment of the present disclosure, the preset intersection-and-union ratio threshold is a value greater than or equal to 0, which can be set according to actual needs, and the embodiment of the present disclosure does not limit this. For example, the preset intersection-and-union ratio threshold can be 0.3, so that for each focus area in the current video frame sequence, if there is a historical focus area in the historical video frame sequence with a target identification intersection-and-union ratio greater than or equal to 0.3 with the focus area, it means that there is a matching focus area in the historical video frame sequence.

[0091] S203: When the intersection-over-union ratios of all target identifiers are less than a preset intersection-over-union ratio threshold, it is determined that there is no matching focus area in the historical focus areas.

[0092] In the disclosed embodiment, for each focus area in the current video frame, when the electronic device determines that the target identification intersection-and-union ratios between all historical focus areas in the historical video frame sequence and the focus area are all less than the preset intersection-and-union ratio threshold, it indicates that all the obtained target identification intersection-and-union ratios are less than the preset intersection-and-union ratio threshold, and thus it can be determined that there is no matching focus area in the historical video frame sequence. For example, when the preset intersection-and-union ratio threshold is 0.3, for each focus area in the current video frame sequence, when the target identification intersection-and-union ratios between all historical focus areas in the historical video frame sequence and the focus area are all less than 0.3, it indicates that there is no matching focus area corresponding to the focus area in the historical video frame sequence.

[0093] In some embodiments, Figure 7 is an optional flow chart of the identification method provided by the embodiment of the present disclosure. The above S103 can also be implemented through S1034-S1035. Figure 7 Take this as an example to illustrate.

[0094] S1034. Obtain historical focus regions of a historical video frame sequence, and select, from the historical video frame sequence, a video frame that contains at least one focus region and whose detection order is closest to the current video frame as the historical video frame.

[0095] In the disclosed embodiment, the electronic device may first determine the detection order between each video frame in the historical video frame sequence, such as which video frame is the first detected video frame and which video frame is the second detected video frame, etc., and based on the obtained detection order between each video frame in the historical video frame sequence, determine from the historical video frame sequence a video frame that contains at least one focus area and whose detection order is closest to the current video frame, and use the selected video frame as the historical video frame and the focus area in the historical video frame as the historical focus area to determine the area category of each focus area in the current video frame.

[0096] In some embodiments, each video frame has a frame number, and the electronic device usually detects multiple video frames according to the order of the frame numbers from large to small; therefore, the electronic device can select a video frame with the smallest difference between the frame number of the current video frame and the frame number of the current video frame as the historical video frame according to the frame numbers between the video frames in the historical video frame sequence. Exemplarily, the historical video frame can be the previous frame of the current video frame, and the previous frame contains the historical focus area.

[0097] In the disclosed embodiment, a video frame that includes a focus area and whose detection order is closest to the current video frame is selected as a historical video frame, and the focus area in the historical video frame is used as the historical focus area to determine the region category of the focus area in the current video frame. This can well determine the correlation between each focus area in the current video frame and the historical focus area, which is beneficial to tracking the same focus area, so that the determined region category of the focus area in the current video frame is more accurate.

[0098] S1035 . Determine the region category of the focus region according to whether there is a matching focus region in the historical focus regions of the historical video frames.

[0099] In the disclosed embodiment, when the electronic device obtains at least one historical focus area, it can first determine whether there are any matching focus areas in the at least one historical focus area, and determine the area category of each focus area in the current video frame based on whether there are any matching focus areas, thereby obtaining the area categories of all focus areas in the current video frame.

[0100] In some embodiments, the above S1033 may be implemented by S301 or S302;

[0101] S301: When there is a matching attention area in the historical attention areas, use the area category of the matching attention area as the area category of the attention area.

[0102] In the embodiment of the present disclosure, for any focus area in the current video frame, when the electronic device determines that there is at least one matching focus area for the focus area in the historical focus area, the region category of the at least one focus area can be used as the region category of the focus area. For example, for a focus area m1, when it is determined that there is a matching focus area P1 for the focus area m1 in the historical focus area, and the region category of the matching focus area P1 is A, it can be determined that the region category of the focus area m1 is A, that is, it indicates that the focus area m1 belongs to the focus area with region category A. For another example, when it is determined that there are two matching focus areas P1 and P2 for the focus area m1 in the historical focus area, and the region category of the matching focus area P1 is A and the region category of the matching focus area P2 is B, it can be determined that the region category of the focus area m1 is A and B, that is, it indicates that the focus area m1 belongs to both the focus area with region category A and the focus area with region category B.

[0103] S302: When there is no matching focused area in the historical focused areas, a new area category is set for the focused area, and the new area category is different from the area category of the historical focused area.

[0104] In the embodiment of the present disclosure, for any focus area in the current video frame, when the electronic device determines that there is no matching focus area for the focus area in the historical focus areas, the focus area can be determined to be a newly added area, and a new area category that is different from the area category of any historical focus area can be set for the newly added area.

[0105] In some embodiments, for any region of interest in the current video frame, when the electronic device determines that there is no matching region of interest for the region of interest in the historical regions of interest, a new patch ID may be set for the region of interest to indicate that the region category of the region of interest is different from the region categories of all other historical regions of interest. It should be noted that the new patch ID is different from the patch IDs of all the obtained regions of interest.

[0106] It can be understood that, for a video frame sequence to be detected comprising 8 video frames, when the recognition method provided by the embodiment of the present disclosure is adopted and it is determined that there is a region of interest in each video frame, the video frame sequence to be detected can obtain at most 8 different patch IDs, that is, these 8 regions of interest are not the same region, so that 8 trajectory sequences can be obtained correspondingly, and there is only one region of interest image in each trajectory sequence; and, the video frame sequence to be detected can obtain at least 1 patch ID, that is, these 8 regions of interest belong to the same region, so that 1 trajectory sequence can be obtained correspondingly, and the trajectory sequence contains 8 region of interest images.

[0107] In the disclosed embodiment, by determining the region category of each focus region in the current video frame, the correlation between each focus region and the historical focus region can be obtained, so that the focus region image contained in each obtained trajectory sequence is more accurate, which is conducive to the correct identification of the trajectory sequence.

[0108] In some embodiments, in some embodiments, as Figure 8 As shown, the above S104 can also be implemented through S401-S402; Figure 8 Take this as an example to illustrate.

[0109] S401 . From each video frame containing each type of focus region in the historical video frame sequence and the current video frame, extract the focus region and the historical focus region to obtain a focus region image.

[0110] In the disclosed embodiment, when the electronic device obtains a video frame sequence consisting of a historical video frame sequence and a current video frame, it can cut out all the focus areas from each video frame of the video frame sequence that contains each category of focus area, and obtain a focus area image corresponding to each category of focus area.

[0111] Here, each region of interest has region coordinates corresponding to the video frame in which it is located. In some embodiments, for each region of interest, the electronic device can extract only the region of interest corresponding to the region coordinates from the corresponding video frame in which the region of interest is located, thereby obtaining a region of interest image corresponding to the region of interest.

[0112] Exemplarily, in a video frame sequence including 4 video frames, the first video frame includes a focus area m1 of category A1 with area coordinates Z1, and a focus area m2 of category A2 with area coordinates Z2; the second video frame includes a focus area m3 of category A1 with area coordinates Z3; the third video frame includes a focus area m4 of category A1 with area coordinates Z4, and a focus area m5 of category A3 with area coordinates Z5; and the fourth video frame includes a focus area m6 of category A3 with area coordinates Z6. The electronic device can cut out the focus area portion corresponding to the area coordinate from the corresponding video frame according to each area coordinate, so as to obtain a focus area image S1 corresponding to the focus area m1, a focus area image S2 corresponding to the focus area m2, a focus area image S3 corresponding to the focus area m3, a focus area image S4 corresponding to the focus area m4, a focus area image S5 corresponding to the focus area m5, and a focus area image S6 corresponding to the focus area m6.

[0113] In some embodiments, for each region of interest, the electronic device may further extract the image portion including the region of interest from the video frame where the region of interest is located, thereby obtaining an image of the region of interest with a larger range; for example, Fig. 9 As shown, for the fourth video frame, the electronic device can cut out the image portion 4-2, wherein the image portion 4-2 includes a portion of the focus area m6 (such as 4-1 in the figure).

[0114] In some embodiments, for all focus areas of each category, the electronic device can determine a general position information corresponding to all focus areas of the category based on the position information of all focus areas in the corresponding video frames of the video frame sequence, and make the area corresponding to the general position information include any focus area of ​​the category.

[0115] Here, the location information can be regional coordinates. For each type of region of interest, the generalized location information can be the union of the regional coordinates corresponding to the regions of interest of that type, for example, the smallest union. For example, among all the regions of interest of class A1, the regional coordinates of the first region of interest m11 in the first video frame are (x11, y11), (x12, y12), and the regional coordinates of the second region of interest m12 in the third video frame are (x21, y21), (x22, y22). And when x11 < x12 < x21 < x22 and y21 < y22 < y11 < y12, the smallest coordinate union of all the regional coordinates corresponding to all the regions of interest of class A1 is: (x11, y21), (x22, y12); where the region corresponding to this smallest coordinate union in the first video frame contains the first region of interest m11, and the region corresponding to this smallest coordinate union in the third video frame contains the second region of interest m12. For example, Fig.10 shows the region 10-1 corresponding to the smallest coordinate union between the regional coordinates (x11, y11)(x12, y12) in the first video frame and the regional coordinates (x21, y21), (x22, y22) in the third video frame. Among them, region 10-1 contains region 10-2 corresponding to the regional coordinates (x11, y11)(x12, y12), and region 10-3 corresponding to the regional coordinates (x21, y21), (x22, y22).

[0116] S402. Obtain a trajectory sequence corresponding to each type of region of interest based on the region-of-interest images; where, when any region of interest has at least two region types, in the trajectory sequences corresponding to each type of region of interest among the at least two region types, each contains a region-of-interest image corresponding to the any region of interest.

[0117] In the embodiments of the present disclosure, the electronic device can determine the trajectory sequence corresponding to each type of region of interest according to all the obtained region-of-interest images.

[0118] When any region of interest has one region type, the electronic device can determine the trajectory sequence corresponding to each type of region of interest according to all the obtained region-of-interest images. For example, continuing with the above example for illustration, the electronic device can obtain the trajectory sequence corresponding to all the regions of interest (m1, m3, m4) of class A1, the trajectory sequence corresponding to all the regions of interest (m2) of class A2, and the trajectory sequence corresponding to all the regions of interest (m5, m6) of class A3 according to these region-of-interest images.

[0119] In the case where any region of interest has two region categories, the electronic device can also determine the trajectory sequence corresponding to each category of the region of interest based on all obtained region of interest images. For example, in the case where the region of interest m1 has region category A1 and region category A2, all regions of interest of category A1 are m1, m3, and m4; all regions of interest of category A2 are m1 and m2; all regions of interest of category A3 are m5 and m6; thus, the electronic device can obtain the trajectory sequence corresponding to all regions of interest (m1, m3, m4) of category A1, the trajectory sequence corresponding to all regions of interest (m1, m2) of category A2, and the trajectory sequence corresponding to all regions of interest (m5, m6) of category A3.

[0120] In some embodiments, when the electronic device obtains the area of ​​interest images, it can classify and sort them according to the area category of the corresponding area of ​​interest, so as to obtain the trajectory sequence corresponding to each category of the area of ​​interest by classifying and sorting the area of ​​interest images belonging to different area categories. In some embodiments, for the area of ​​interest images of each area category, the electronic device can sort the area of ​​interest images of this category according to the time sequence (i.e., the detection order) between the video frames where the corresponding area of ​​interest is located. For example, the area of ​​interest images of this category can be sorted in the same order according to the time sequence between the video frames where the corresponding area of ​​interest is located; in this way, the appearance order of the target objects contained in the area of ​​interest images is the same as the actual appearance order, so that when the obtained trajectory sequence is subsequently recognized, the recognition result obtained is more accurate.

[0121] For example, among the obtained focus area images of all area categories, among the four focus area images corresponding to all focus areas of category A1, the focus area image G1 is located in the first video frame, the focus area image G2 is located in the fourth video frame, the focus area image G3 is located in the second video frame, and the focus area image G4 is located in the seventh video frame. The electronic device can sort G1 to G4 in the same order according to the time sequence of: the first video frame, the second video frame, the fourth video frame, and the seventh video frame to obtain the sorting results: focus area image G1, focus area image G3, focus area image G2, and focus area image G4; and use the sorted focus area images as category A1. The trajectory sequence corresponding to all the focus areas of B1 is obtained; and among the focus area images of all area categories, among the three focus area images corresponding to all the focus areas of B1 category, the focus area image G1 is located in the first video frame, the focus area image G5 is located in the third video frame, and the focus area image G6 is located in the fourth video frame. The electronic device can sort G1, G5 and G6 in the same order according to the time sequence of the first video frame, the third video frame, and the fourth video frame to obtain the sorting result: the focus area image G1, the focus area image G5, and the focus area image G6; and use the sorted focus area images as the trajectory sequence corresponding to all the focus areas of B1 category.

[0122] The following will illustrate the identification solution provided by the embodiment of the present disclosure through an application scenario embodiment.

[0123] S1, performing target detection on the current video frame, and when it is determined that there is a real person in the current video frame, determining the bounding box area of ​​each real person and the person ID;

[0124] S2. When there are at least two real people in the current video frame, the area of ​​the bounding box region of each real person is proportionally enlarged by 1.5 times to obtain an enlarged area of ​​each real person, and the area intersection and union ratio between any two enlarged areas is calculated, and when the area intersection and union ratio is greater than 0, it is determined that the any two enlarged areas are matched; thereby, the two enlarged areas with the most other matched enlarged areas in the current video frame are determined, and the two enlarged areas are used as the two key areas of the current video frame;

[0125] S3, the area of ​​the bounding box area corresponding to each key area in the two key areas is proportionally enlarged by 2 times to obtain the focus area corresponding to each key area, that is, to obtain two focus areas in the current video frame;

[0126] S4. In the case that a previous video frame of the current video frame includes at least one focus region, the previous video frame is used as a historical video frame, and at least one key region in the previous video frame is used as at least one historical key region, and at least one focus region in the previous video frame is used as at least one historical focus region, wherein each historical focus region has a region ID (patch ID);

[0127] S5. For each region of interest in the current video frame, calculate the intersection-and-union ratio of the character ID between the region of interest and each historical region of interest, and if there is at least one historical region of interest whose intersection-and-union ratio of the character ID with the region of interest is greater than 0.3, use the at least one historical region of interest as at least one matching region of interest, and use all the patch IDs of the at least one matching region of interest as the patch ID of the region of interest, to obtain at least one patch ID of the region of interest; and if there is no matching region of interest in at least one historical region of interest, set a new patch ID for the region of interest, and the new patch ID is different from all previous patch IDs;

[0128] S6. When the detection time of the video frame reaches 3 seconds, the current video frame detected at the 3rd second and the historical video frame sequence of the current video frame are taken as the video frame sequence (i.e., all the video frames detected within the 3 seconds are taken as the video frame sequence), and according to the position coordinates of the focus area of ​​each category in the video frame corresponding to the focus area of ​​the video sequence, a generalized position coordinate corresponding to the focus area of ​​the category is determined; the area corresponding to the generalized position coordinate includes any focus area of ​​the focus area of ​​the category;

[0129] S7, from each video frame in the video frame sequence that contains the region of interest of the category, extracting a region corresponding to the generalized position coordinates to obtain a region of interest image corresponding to the region of interest of the category;

[0130] S8. For the region of interest images corresponding to the region of interest of the category, all the region of interest images are sorted according to the size relationship between the frame numbers of the video frame where each region of interest is located and other video frames, to obtain a trajectory sequence corresponding to the region of interest of the category; in the case where any region of interest has at least two region categories, the trajectory sequence corresponding to the region of interest of each category in the at least two region categories includes a region of interest image corresponding to the any region of interest;

[0131] S9. Use a classification network to classify and identify the trajectory sequence corresponding to the focus area of ​​this category, and determine whether the person corresponding to the trajectory sequence has engaged in fighting or other behaviors.

[0132] S10. When it is determined that there is no real person in the current video frame, a preset size area with the center of the current video frame as the area center is used as a regular focus area, and the target detection is continued for the next video frame. When the detection time of the video frame reaches 3 seconds and there is no target object in all the video frames detected within these 3 seconds, the image part corresponding to the area coordinates of the regular focus area is cut out from the video frame where each regular focus area is located to obtain a regular focus area image corresponding to each regular focus area; and all the regular focus area images are sorted according to the order in which the regular focus areas corresponding to each regular focus area image are obtained, and the sorted regular focus area images are used as the regular trajectory image sequence.

[0133] S11. Using a classification network, classify and identify the regular trajectory image sequence, and determine the behavior of the object corresponding to the regular trajectory image sequence.

[0134] The present disclosure also provides an identification device, Fig.11 A schematic diagram of the structure of the identification device provided in the embodiment of the present disclosure; Fig.11 As shown, the recognition device 1 includes: a detection unit 10, which is used to perform target detection on the current video frame and determine the bounding box area of ​​at least one target object; a determination unit 20, which is used to determine the focus area of ​​the current video frame according to the bounding box area; obtain the historical focus area of ​​the historical video frame sequence, and determine the category of the focus area according to the intersection and union ratio between the focus area and the historical focus area; the historical video frame sequence is a video frame sequence that has been detected before the current video frame; a segmentation unit 30, which is used to determine the trajectory sequence corresponding to each type of focus area from the historical video frame sequence and the current video frame; and a recognition unit 40, which is used to identify the trajectory sequence corresponding to each type of focus area to obtain a corresponding recognition result.

[0135] In some embodiments of the present disclosure, the determination unit 20 is further used to determine whether there is a matching focus area in the historical focus area based on the intersection-over-union ratio between the focus area and the historical focus area; and determine the category of the focus area based on whether there is the matching focus area in the historical focus area.

[0136] In some embodiments of the present disclosure, the determination unit 20, when the matching area of ​​interest exists in the historical area of ​​interest, uses the area category of the matching area of ​​interest as the area category of the area of ​​interest; and / or, when the matching area of ​​interest does not exist in the historical area of ​​interest, sets a new area category for the area of ​​interest, and the new area category is different from the area category of the historical area of ​​interest.

[0137] In some embodiments of the present disclosure, the segmentation unit 30 is further used to extract the focus area and the historical focus area from each video frame containing each type of focus area in the historical video frame sequence and the current video frame to obtain a focus area image; based on the focus area image, obtain a trajectory sequence corresponding to each type of focus area; wherein, in the case where any focus area has at least two area categories, the trajectory sequence corresponding to each type of focus area in the at least two area categories includes a focus area image corresponding to any focus area.

[0138] In some embodiments of the present disclosure, each bounding box area corresponds to a target identifier; the intersection-and-union ratio includes: a target identifier intersection-and-union ratio; the determination unit 20 is also used to determine the target identifier intersection-and-union ratio between each focus area and each historical focus area; when at least one target identifier intersection-and-union ratio is greater than or equal to a preset intersection-and-union ratio threshold, it is determined that there is at least one matching focus area in the historical focus area; when all target identifier intersection-and-union ratios are less than the preset intersection-and-union ratio threshold, it is determined that there is no matching focus area in the historical focus area.

[0139] In some embodiments of the present disclosure, the determination unit 20 is also used to determine, for any area of ​​interest and any historical area of ​​interest, a first target identifier corresponding to each bounding box area contained in any area of ​​interest, to obtain at least one first target identifier; and determine a second target identifier corresponding to each bounding box area contained in any historical area of ​​interest, to obtain at least one second target identifier; determine a first number of identical target identifiers and a second number of different target identifiers in the at least one first target identifier and the at least one second target identifier; determine the sum of the first number and the second number, and use the ratio of the first number to the sum as the target identifier intersection-union ratio between any area of ​​interest and any historical area of ​​interest.

[0140] In some embodiments of the present disclosure, the determination unit 20 is further used to select a video frame from the historical video frame sequence that contains at least one focus area and whose detection order is closest to the current video frame as a historical video frame; and determine the category of the focus area based on whether the matching focus area exists in the historical focus areas of the historical video frame.

[0141] In some embodiments of the present disclosure, the target object includes a real person; the recognition unit 40 is further used to use a classification network to classify and recognize the trajectory sequence corresponding to each type of focus area, so as to obtain the person behavior corresponding to the trajectory sequence corresponding to each type of focus area.

[0142] The present disclosure also provides an electronic device, Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure, such as Fig.12 As shown, the electronic device 2 includes: a memory 21 and a processor 22, wherein the memory 21 and the processor 22 are connected via a communication bus 23; the memory 21 is used to store an executable computer program; the processor 22 is used to implement the method provided by the embodiment of the present disclosure when executing the executable computer program stored in the memory 21, for example, the identification method provided by the embodiment of the present disclosure.

[0143] The embodiment of the present disclosure provides a computer-readable storage medium storing a computer program for causing the processor 22 to execute and implement the method provided by the embodiment of the present disclosure, for example, the identification method provided by the embodiment of the present disclosure.

[0144] In some embodiments of the present disclosure, the storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0145] In some embodiments of the present disclosure, executable instructions may be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines or other units suitable for use in a computing environment.

[0146] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0147] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0148] In summary, target detection is performed on the current video frame, and when the target object is detected, the bounding box area of ​​each target object is determined, and at least one key area of ​​the current video frame is determined from all enlarged areas obtained by enlarging the area of ​​the bounding box area of ​​each target object, wherein each enlarged area is an area obtained by enlarging the area of ​​a bounding box area, and each key area corresponds to a focus area, and each focus area at least includes a key area corresponding to it; and according to whether there is a matching focus area in the historical video frame, the area category of each focus area of ​​the current video frame is determined, wherein the intersection-and-union ratio between each matching focus area and the focus area satisfies a preset intersection-and-union ratio threshold, and the historical video frame is a video frame that is before the current video frame and has been detected, and each matching focus area is a historical focus area in the historical video frame sequence; from the historical video sequence and the video frame sequence of the current video frame, the trajectory sequence corresponding to each category of the focus area is determined, and the trajectory sequence corresponding to each category of the focus area is identified to obtain the corresponding recognition result. Since, for each video frame containing the target object, the key area of ​​the video frame is determined, and the focus area in the video frame that at least contains the key area and has a region category is determined, so the focus area of ​​each category obtained is the important area in the corresponding video frame; and, for the obtained video frame sequence composed of the current video frame and the historical video frame sequence, according to the focus area of ​​each category contained in the video frames of the video frame sequence, the trajectory sequence of the focus area of ​​each category corresponding to the video frame sequence is obtained, so the trajectory sequence corresponding to the focus area of ​​each category obtained is composed of important areas belonging to the same area in the video frame sequence; therefore, when identifying the trajectory sequence of the focus area of ​​each category, compared with identifying the entire image of each video frame in the video frame sequence, the interference of background factors in the image on the identification is reduced, and the calculation amount and recognition range during identification are reduced, thereby improving the recognition efficiency and improving the accuracy of the obtained recognition result.

[0149] The above description is only an embodiment of the present disclosure and is not intended to limit the protection scope of the present disclosure. Any modification, equivalent replacement and improvement made within the spirit and scope of the present disclosure are included in the protection scope of the present disclosure.

Claims

1. A recognition method, characterized in that: include: Performing target detection on the current video frame to determine a bounding box area of ​​at least one target object; Determine a focus area of ​​the current video frame according to the bounding box area; Acquire a historical focus region of a historical video frame sequence, and determine the category of the focus region according to an intersection-and-union ratio between the focus region and the historical focus region; the historical video frame sequence is a video frame sequence that is before the current video frame and has been detected; Determine, from the historical video frame sequence and the current video frame, a trajectory sequence corresponding to each type of focus area; Identify the trajectory sequence corresponding to each type of focus area to obtain a corresponding recognition result; The determining the category of the focus area according to the intersection-over-union ratio between the focus area and the historical focus area includes: Each bounding box area corresponds to a target identifier; the intersection-over-union ratio includes the intersection-over-union ratio of the target identifier; Determine a first target identifier corresponding to each bounding box area included in any focus area to obtain at least one first target identifier; and determine a second target identifier corresponding to each bounding box area included in any historical focus area to obtain at least one second target identifier; Determining a first number of identical target identifiers and a second number of different target identifiers in the at least one first target identifier and the at least one second target identifier; Determine a sum of the first number and the second number, and use a ratio of the first number to the sum as a target identifier intersection-and-union ratio between any of the focus regions and any of the historical focus regions; When the intersection-over-union ratio of at least one target identifier is greater than or equal to a preset intersection-over-union ratio threshold, determining that there is at least one matching focus area in the historical focus area; When the intersection-over-union ratios of all target identifiers are less than the preset intersection-over-union ratio threshold, determining that the matching focus area does not exist in the historical focus area; According to whether the matching focus area exists in the historical focus areas, the category of the focus area is determined.

2. The method according to claim 1, characterized in that The determining the category of the focus area according to whether the matching focus area exists in the historical focus area includes: If the matching region of interest exists in the historical regions of interest, using the region category of the matching region of interest as the region category of the region of interest; and / or, In the case that the matching region of interest does not exist in the historical regions of interest, a newly added region category is set for the region of interest, and the newly added region category is different from the region category of the historical regions of interest.

3. The method according to claim 1 or 2, characterized in that: The step of determining a trajectory sequence corresponding to each type of focus area from the historical video frame sequence and the current video frame includes: From each video frame containing each type of focus region in the historical video frame sequence and the current video frame, extract the focus region and the historical focus region to obtain a focus region image; Based on the area of ​​interest image, a trajectory sequence corresponding to each type of area of ​​interest is obtained; wherein, in the case where any area of ​​interest has at least two area categories, the trajectory sequence corresponding to each type of area of ​​interest in the at least two area categories includes an area of ​​interest image corresponding to any area of ​​interest.

4. The method according to claim 1, characterized in that The determining the category of the focus area according to whether the matching focus area exists in the historical focus area includes: From the historical video frame sequence, select a video frame that contains at least one focus area and whose detection order is closest to the current video frame as a historical video frame; The category of the attention region is determined according to whether the matching attention region exists in the historical attention regions of the historical video frame.

5. The method according to claim 1, characterized in that The target object includes a real person; the identifying the trajectory sequence corresponding to each type of focus area to obtain the corresponding recognition result includes: A classification network is used to classify and identify the trajectory sequence corresponding to each type of focus area, so as to obtain the character behavior corresponding to the trajectory sequence corresponding to each type of focus area.

6. An identification device, characterized in that: include: A detection unit, configured to perform target detection on a current video frame and determine a bounding box area of ​​at least one target object; a determination unit, configured to determine the focus area of ​​the current video frame according to the bounding box area; obtain the historical focus area of ​​a historical video frame sequence, and determine the category of the focus area according to the intersection-and-union ratio between the focus area and the historical focus area; the historical video frame sequence is a video frame sequence that is before the current video frame and has been detected; A segmentation unit, used to determine a trajectory sequence corresponding to each type of focus area from the historical video frame sequence and the current video frame; An identification unit, used to identify the trajectory sequence corresponding to each type of focus area and obtain a corresponding identification result; The determining unit is further used to determine a first target identifier corresponding to each bounding box area included in any focus area, and obtain at least one first target identifier; and determining a second target identifier corresponding to each bounding box area included in any historical focus area to obtain at least one second target identifier; Determine a first number of identical target identifiers and a second number of different target identifiers in the at least one first target identifier and the at least one second target identifier; determine a sum of the first number and the second number, and use a ratio of the first number to the sum as a target identifier intersection-and-union ratio between any of the regions of interest and any of the historical regions of interest; When at least one target identification intersection-and-union ratio is greater than or equal to a preset intersection-and-union ratio threshold, it is determined that at least one matching area of ​​interest exists in the historical area of ​​interest; when all target identification intersection-and-union ratios are less than the preset intersection-and-union ratio threshold, it is determined that there is no matching area of ​​interest in the historical area of ​​interest; According to whether the matching focus area exists in the historical focus area, the category of the focus area is determined; wherein each bounding box area corresponds to a target identifier; and the intersection-over-union ratio includes the intersection-over-union ratio of the target identifier.

7. An electronic device, characterized in that: include: a memory for storing executable computer programs; A processor, configured to implement the method according to any one of claims 1 to 5 when executing the executable computer program stored in the memory.

8. A computer-readable storage medium, characterized in that: A computer program is stored, which is used to cause a processor to execute and implement the method according to any one of claims 1 to 5.

Images